Systems and methods for automated analysis, screening, and reporting of group performance

ABSTRACT

A method for quantifying performance of a group includes obtaining, for the group, historical group performance data, historical group factor exposure data for a plurality of predefined factors, and historical factor performance data for the plurality of predefined factors. The method includes generating historical group residual performance data for the group in accordance with the historical group performance data, the historical group factor exposure data for the plurality of predefined factors, and the historical factor performance data for the plurality of predefined factors. A respective entry in the historical group residual performance data represents member selection performance associated with the group. The method includes storing the historical group residual performance data for the group; and providing one or more values that represent the historical group residual performance data.

RELATED APPLICATIONS

This application is a continuation-in-part of U.S. patent applicationSer. No. 14/449,870, filed Aug. 1, 2014, which claims priority to U.S.Provisional Patent Application Ser. No. 61/861,236, filed Aug. 1, 2013and U.S. Provisional Patent Application Ser. No. 61/884,950, filed Sep.30, 2013, all of which are incorporated by reference herein in theirentirety.

TECHNICAL FIELD

The disclosed embodiments relate generally to performance monitoring ofan entity (e.g., a machine). More particularly, the disclosedembodiments relate to apparatus and process for performance monitoringof a group of entities (e.g., a group of machines).

BACKGROUND

Increasingly more people analyze performance of entities (e.g.,machines) using quantitative tools, computer methods, and computersystems. The performance of an entity is frequently expressed as achange in the value of a particular property of an entity over a certainperiod (e.g., failure rate of machines).

However, with respect to a group of entities (e.g., a group ofmachines), it is difficult to accurately represent the performance ofthe group using conventional tools. Therefore, there is a need for a newmethod and system for analyzing performances of groups of entities.

SUMMARY

In particular, conventional tools typically measure the performance of agroup based on whether a particular property of the group has increasedor decreased. However, the conventional tools do not indicate why thevalue of a particular property of the group has increased or decreased.This has led some agents to take unreasonable actions to manage thevalue of a particular property of the group (e.g., making exclusive useof new machines that have a low chance of failure when use of oldermachines may be more economical, making use of mechanical proceduresthat have a low chance of failure but may be more costly), becauseconventional tools do not adequately indicate the sources of theperformance of those agents, or groups of entities managed by thoseagents. In addition, conventional tools do not identify whether thevalue of a group of entities increased due to an intrinsic property ofthis group, the skill of an agent managing the group, or due to a pureluck.

A number of embodiments (e.g., of systems, and methods of operating suchsystems) that overcome the limitations and disadvantages described aboveare presented in more detail below. These embodiments provide methods,systems, and graphical user interfaces (GUIs) for analyzing performancesof groups.

As described in more detail below, some embodiments involve acomputer-implemented method that includes receiving, at a computersystem with one or more processors and memory, a request to quantifygroup residual performance associated with a first group; obtaining, atthe computer system, historical group performance data for the firstgroup; obtaining, at the computer system, historical group factorexposure data for a plurality of predefined factors for the first group;and obtaining, at the computer system, historical factor performancedata for the plurality of predefined factors for the first group. Themethod also includes generating, at the computer system, historicalgroup residual performance data for the first group in accordance withthe historical group performance data for the first group, thehistorical group factor exposure data for the plurality of predefinedfactors for the first group, and the historical factor performance datafor the plurality of predefined factors for the first group. Arespective entry in the historical group residual performance datarepresents member selection performance associated with the first group(e.g., for a particular time period). The method also includes storing,at the computer system, the historical group residual performance datafor the first group; and providing one or more values that represent thehistorical group residual performance data.

A computer system includes one or more processors for executing programsand memory storing one or more programs for execution by the one or moreprocessors. The one or more programs include instructions executed bythe one or more processors so as to perform any of the embodiments ofthe aforementioned method.

A computer readable storage medium stores one or more programsconfigured for execution by one or more processors of a computer system.The one or more programs include instructions for performing any of theembodiments of the aforementioned method.

For example, in some embodiments, a computer readable storage medium(e.g., a non-transitory computer readable storage medium) stores one ormore programs for execution by one or more processors of a computersystem. The one or more programs include instructions for receiving, atthe computer system, a request to quantify group residual performanceassociated with a first group; obtaining, at the computer system,historical group performance data for the first group; obtaining, at thecomputer system, historical group factor exposure data for a pluralityof predefined factors for the first group; obtaining, at the computersystem, historical factor performance data for the plurality ofpredefined factors for the first group; generating, at the computersystem, historical group residual performance data for the first groupin accordance with the historical group performance data for the firstgroup, the historical group factor exposure data for the plurality ofpredefined factors for the first group, and the historical factorperformance data for the plurality of predefined factors for the firstgroup, wherein a respective entry in the historical group residualperformance data represents member selection performance associated withthe first group; storing, at the computer system, the historical groupresidual performance data for the first group; and providing one or morevalues that represent the historical group residual performance data.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the aforementioned aspects of theinvention as well as additional aspects and embodiments thereof,reference should be made to the Description of Embodiments below, inconjunction with the following drawings in which like reference numeralsrefer to corresponding parts throughout the figures.

FIG. 1 is a block diagram illustrating a process flow diagram ofhistorical factor exposure and factor performance data estimationcomponents of a computer system in accordance with some embodiments.

FIG. 2 is a block diagram illustrating a process flow diagram ofhistorical group member composition and performance data collectioncomponents of a computer system in accordance with some embodiments.

FIG. 3 is a block diagram illustrating a process flow diagram ofhistorical group and agent performance attribution and skill scoringdata estimation components of a computer system in accordance with someembodiments.

FIG. 4 is a block diagram illustrating a process flow diagram of userand application interfaces to data generated by a computer system inaccordance with some embodiments.

FIG. 5 illustrates an example of a group screening interface inaccordance with some embodiments.

FIG. 6 illustrates an exemplary user interface in accordance with someembodiments.

FIG. 7 is a scatter plot that illustrates selected performanceindicators of groups in accordance with some embodiments.

FIG. 8 is a heat map that illustrates a selected performance indicatorof groups in accordance with some embodiments.

FIG. 9 is a correlation chart that illustrates similarity of groupsbased on a selected performance indicator in accordance with someembodiments.

FIG. 10 is a dendrogram that illustrates clustering of groups based on aselected performance indicator in accordance with some embodiments.

FIG. 11 is a block diagram illustrating an exemplary distributedcomputer system in accordance with some embodiments.

FIG. 12 is a block diagram illustrating a computer system in accordancewith some embodiments.

FIGS. 13A-13G are flow charts representing a method of quantifyingperformance of a group in accordance with some embodiments.

FIG. 14 is a conceptual diagram illustrating generation of a highperforming group in accordance with some embodiments.

FIGS. 15A-15B are graphs illustrating example distributions ofreliability values in accordance with some embodiments.

FIGS. 16A-16B are graphs illustrating example autocorrelation valuesassociated with performance of a group in accordance with someembodiments.

FIGS. 17A-17H are flow charts representing a method of quantifyingperformance of a group in accordance with some embodiments.

Like reference numerals refer to corresponding parts throughout thedrawings.

DESCRIPTION OF EMBODIMENTS

Methods and systems for analyzing performances of groups are described.Reference will be made to certain embodiments, examples of which areillustrated in the accompanying drawings. It will be understood that theembodiments are not intended to limit the scope of claims to theseparticular embodiments alone. On the contrary, the claims are intendedto cover alternatives, modifications and equivalents that are within thespirit and scope of the described principles as defined by the appendedclaims.

Moreover, in the following description, numerous specific details areset forth to provide a thorough understanding of the embodiments.However, it will be apparent to one of ordinary skill in the art thatthe claimed features may be implemented without these particulardetails. In other instances, methods, procedures, components, andnetworks that are well-known to those of ordinary skill in the art arenot described in detail to avoid obscuring aspects of the claimedfeatures.

It will also be understood that, although the terms first, second, etc.may be used herein to describe various elements, these elements shouldnot be limited by these terms. These terms are only used to distinguishone element from another. For example, a first factor could be termed asecond factor, and, similarly, a second factor could be termed a firstfactor, without departing from the scope of the present invention. Thefirst factor and the second factor are both contacts, but they are notthe same factors.

The terminology used in the description of the embodiments herein is forthe purpose of describing particular embodiments only and is notintended to be limiting of the invention. As used in the description ofthe invention and the appended claims, the singular forms “a,” “an,” and“the” are intended to include the plural forms as well, unless thecontext clearly indicates otherwise. It will also be understood that theterm “and/or” as used herein refers to and encompasses any and allpossible combinations of one or more of the associated listed items. Itwill be further understood that the terms “comprises” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

As used herein, the term “if” may be construed to mean “when” or “upon”or “in response to determining” or “in response to detecting,” dependingon the context. Similarly, the phrase “if it is determined” or “if [astated condition or event] is detected” may be construed to mean “upondetermining” or “in response to determining” or “upon detecting (thestated condition or event)” or “in response to detecting (the statedcondition or event),” depending on the context.

Components

FIG. 1 illustrates a process flow diagram of historical factor exposureand factor performance data estimation components of a computer systemin accordance with some embodiments. The following are illustrated inFIG. 1:

-   -   Member data 102 processed by a web crawler, document downloader        and converter 104;    -   Web crawler, document downloader and converter 104 processing        member data 102 to generate historical member performance and        fundamental data 110;    -   Third party member data 106 processed by an external member data        query and conversion module 108;    -   External member data query and conversion module 108 processing        third party member data 106 to generate historical member        performance and fundamental data 110;    -   Historical member performance and fundamental data 110 populated        by web crawler, document downloader and converter 104 and        external member data query and conversion module 108, used by        factor performance estimator 114 and member factor exposure        estimator 118;    -   Factor model specification data 112 used by factor performance        estimator 114 and member factor exposure estimator 118;    -   Factor performance estimator 114, processing historical member        performance and fundamental data 110 according to factor model        specification data 112 to generate historical factor performance        data 116;    -   Historical factor performance data 116, generated by factor        performance estimator 114 and used by member factor exposure        estimator 118;    -   Member factor exposure estimator 118, processing historical        member performance and fundamental data 110 as well as        historical factor performance data 116 according to factor model        specification data 112 to generate historical member factor        exposure data 120; and    -   Historical member factor exposure data 120, generated by member        factor exposure estimator 118.

FIG. 2 illustrates a process flow diagram of historical group membercomposition and performance data collection components of a computersystem in accordance with some embodiments. The following areillustrated in FIG. 2:

-   -   Government filings 202, processed by web crawler, document        downloader, and parser 212;    -   Third party historical group data 204, processed by group data        query and conversion engine 214;    -   Third party and agent group data 206, processed by group data        query and conversion engine 214;    -   Historical member identification data 208, used by web crawler,        document downloader, and parser 212, group data query and        conversion engine 214 and historical group management user        interface (UI) 218, which updates historical group member        composition and performance data 220 in some embodiments;    -   Users 210 (which is included for illustration purposes only and        does not form any part of the computer system), accessing        historical group management user interface 218;    -   Web crawler, document downloader, and parser 212 using        historical member identification data 208 to process government        filings 202 and generate historical group member composition and        performance data 220;    -   Group data query and conversion engine 214, using historical        member identification data 208 to process third party historical        group data 204 and third party and agent group data 206 to        generate historical group member composition and performance        data 220;    -   Historical group management user interface 218, using and        updating historical member identification data 208 and receiving        inputs from, and providing information to, users 210 to generate        historical group member composition and performance data 220;        and    -   Historical group member composition and performance data 220,        generated by web crawler, document downloader, and parser 212,        group data query and conversion engine 214 and/or historical        group management user interface 218.

FIG. 3 illustrates a process flow diagram of historical group and agentperformance attribution and skill scoring data estimation components ofa computer system in accordance with some embodiments. The following areillustrated in FIG. 3:

-   -   Historical member factor exposure data 120, processed by group        factor exposure estimator 310;    -   Factor model specification data 304, used by group factor        exposure estimator 310;    -   Historical group composition and performance data 220, used by        group factor exposure estimator 310 and group factor performance        attribution engine 314;    -   Historical factor performance data 116, used by group factor        performance attribution engine 314 and statistical analysis and        scoring engine 324;    -   Group factor exposure estimator 310, generating historical group        factor exposure data 312 using historical member factor exposure        data 120, factor model specification data 304 and historical        group composition and performance data 220;    -   Historical group factor exposure data 312, generated by group        factor exposure estimator 310 and used by group factor        performance attribution engine 314;    -   Group factor performance attribution engine 314, using        historical group factor exposure data 312, historical group        composition and performance data 220, and historical factor        performance data 116 to calculate historical group performance        attribution data 316;    -   Historical group performance attribution data 316, calculated by        group factor performance attribution engine 314 and used by        aggregator of historical agent group association and performance        data 320 and statistical analysis and scoring engine 324;    -   Historical agent group association data 318, used by aggregator        of historical agent group association and performance data 320;    -   Aggregator of historical agent group association and performance        data 320, using historical agent group association data 318 and        historical group performance attribution data 316 to generate        historical agent performance attribution data 322;    -   Historical agent performance attribution data 322, generated by        aggregator of historical agent group association and performance        data 320 and used by statistical analysis and scoring engine        324;    -   Statistical analysis and scoring engine 324, using historical        factor performance data 116, historical group performance        attribution data 316, and/or historical agent performance        attribution data 322 to generate historical group skill scoring        data 326 and/or historical agent skill scoring data 328;    -   Historical group skill scoring data 326, generated by        statistical analysis and scoring engine 324; and    -   Historical agent skill scoring data 328, generated by        statistical analysis and scoring engine 324.

FIG. 4 illustrates a process flow diagram of user and applicationinterfaces to data generated by a computer system in accordance withsome embodiments. The following are illustrated in FIG. 4:

-   -   Historical group performance attribution data 316, historical        agent performance attribution data 322, historical group skill        scoring data 326, historical agent skill scoring data 328 and        other miscellaneous group and agent data 410 (collectively,        system data), accessed by reporting engine 412, data query        engine 414, and screening engine 416;    -   Reporting engine 412, processing the system data to populate        group performance and evaluation report database 418 and agent        performance and evaluation report database 420, accessed by        reporting user interface 422 to define and provide custom        reports;    -   Data query engine 414, used by data query user interface 424 and        APIs and web services 428 to access the system data;    -   Screening engine 416, used by screening user interface 426 and        APIs and web services 428 to access the system data and filter        data for groups and agents meeting various criteria;    -   Group performance and evaluation report database 418, populated        by reporting engine 412 and used by report user interface 422        and APIs and web services 428 allowing saving and caching of        reports;    -   Agent performance and evaluation report database 420, populated        by reporting engine 412 and used by report user interface 422        and APIs and web services 428 allowing saving and caching of        reports;    -   Report user interface 422, providing users 430 access to reports        from group performance and evaluation report database 418 and        agent performance and evaluation report database 420 and        receiving inputs from users 430 to define custom reports;    -   Data query user interface 424, providing users 430 access to        data query engine 414;    -   Screening user interface 426, providing users 430 access to        screening engine 416;    -   APIs and web services 428, providing third party applications        432 access to group performance and evaluation report database        418, agent performance and evaluation report database 420, data        query engine 414, and screening engine 416;    -   Users 430 (which is included for illustration purposes only and        does not form any part of the computer system), accessing report        user interface 422, data query user interface 424, and screening        user interface 426 and receiving data therefrom; and    -   Third party applications 432, accessing APIs and web services        428 and receiving data therefrom.

Although FIGS. 1-4 illustrate particular aspects of a computer systemwith certain components, a person having ordinary skill in the art wouldunderstand that the computer system may include a subset, or a superset,of the illustrated components.

Operations

In some embodiments, a computer system operates using a set of factormodels. The set of factor models includes historical factor performancedata 116 and historical member factor exposure data 120. Historicalfactor performance data 116 may be stored as time series of performancesand the statistics associated with such performance data for each modelfactor provided. Historical member factor exposure data 120 may bestored as time series of exposures for each member supported to eachmodel factor provided and applicable to the member.

In some embodiments, factor model data 116 and 120 is generated asfollows. Member data 102 and 106 are queried, processed and converted bymodules 104 and 108 to produce historical member performance andfundamental data 110. Historical member data 110 contains all theinformation necessary to estimate factor performances 116 and historicalmember factor exposure data 120 for the models defined by factor modelspecification data 112. Factor model specification data 112 define howfactor performances and exposures are to be calculated for each model.For example, in some implementations, a factor model specification infactor model specification data 112 defines that the model consists of asingle factor. Factor performance for a given period is obtained bycalculating a composite performance of all members using weighting. Inthis case, database 110 is queried by factor performance estimator 114for performance and weights for members in each estimation period.Factor performance estimator 114 combines historical member performancedata 110 according to the specification in factor model specificationdata 112 to produce historical factor performance data 116. Forinstance, a factor model specification in factor model specificationdata 112 may include sector factors. A sector factor corresponds to aplurality of members. In some cases, factor performance estimator 114calculates weighted average historical performances for members in thissector using historical member performance and fundamental data 110.Member factor exposure estimator 118 combines historical memberperformance data 110 and, for models that require it, historical factorperformance data 116, according to a specification in factor modelspecification data 112 to produce historical member factor exposure data120. In case statistical factor models are used, this process typicallyinvolves performing a regression analysis for member performancesagainst factor performances. The regression coefficients from theregression analysis yield values stored in member factor exposure data120.

In some embodiments, a computer system processes collection ofhistorical group member composition and performance data 220. Historicalgroup member composition and performance data 220 may be stored as atime series of group composition and performances or a collection oftables or documents containing group composition and performance datafor each observation period. Historical group member composition data220 may be generated from a query and processing of regulatory filings202 using web crawler, downloader, and parser 212. Web crawler,downloader, and parser 212 may be implemented in a high-level extractionand reporting language such as Perl or Python. Web crawler, downloader,and parser 212 downloads the index of regulatory filings for a group,downloads all the individual filings, parses them to extract the groupinformation, and updates historical group member composition andperformance data 220. Third party historical group data 204 and thirdparty and agent group data 206 may be converted by module 214 to matchthe format of historical group member compositions and performance data220. This allows integration with existing third party and agent grouptracking and management systems. Users 210 can use management interface218 to directly enter and manage historical group information 220. Thismay be used by users to analyze sub-components of their overall group ormodel groups. Historical member identification data 208 is used byparser 212, converter 214, and interface 218 to allow identification of,and reference to, members using multiple symbols, and names, since amember may have aliases in government filings and composition reports.

In some embodiments, the analysis, reporting, and screening of group andagent performance relies on historical group performance attributiondata 316 and historical agent performance attribution data 322. For agiven factor model specification from the collection 304, historicalgroup compositions 220 (also called herein group member exposures) arecombined with historical member factor exposure data 120 (also calledherein member factor exposures) by the group exposure estimator 310 toyield historical group factor exposure data 312. In some embodiments,group factor exposures or group active factor exposures active exposuresrelative to a group benchmark may be used. Historical group factorexposure data 312 (also called herein group factor exposures) may bestored in a variety of formats, such as time series of exposures foreach group covered by the system to each model factor applicable to themembers in the group, as a collection of vectors of exposures for eachgroup to model factors in a given observation period, or any alternativemethod encompassing the same information.

In some embodiments, historical group factor exposure data 312,historical group composition and performance data 220, and historicalfactor performance data 116 (also called herein factor performances) areprocessed by the performance attribution engine 314 to generatehistorical group performance attribution data 316. In some embodiments,the group performance includes residual performances and static anddynamic performances for all defined factors. In some embodiments, thisdata also includes group allocation performance and group selectionperformance.

In some cases, an agent may be involved in the management of multiplegroups simultaneously and/or over history. An association among an agentand historical groups is stored in historical agent group associationdatabase 318. Database 318 may be implemented as a table with an entryfor each group and agent managing that group over a given period. Datain database 318 may be combined with information in historical groupperformance attribution data 316 by aggregator 320 to yield historicalagent performance attribution data 322. A particular version ofaggregator 320 may be implemented as:

-   -   For each measurement period p and agent M, using the data in        database 318 identify the set of all groups P managed by M        during p.    -   Retrieve the set of performance attribution data for P from data        316 containing all of the defined performance components.    -   Calculate composite performance attribution for M over p as a        by-element weighted-average of all performance attribution data.        An aggregate time series of performance attribution data is        established by linking entries for each period p of an agent's        history. The process results in historical agent performance        attribution data 322 containing aggregate performance        attribution components for each period p. A number of weighting        and linking factors can be used for each period's data such as        group value and equal weighting.

An alternative version of an aggregator 320 operates to construct acomposite group for each measurement period p and perform attribution onthe composite as follows. For each measurement period p and agent M,using data 318, identify the set of all groups managed by M during p.Create a composite group CP as a combination of all groups whereindividual members may be equal-weighted, value-weighted, or use analternative weighting mechanism. Calculate historical agent performanceattribution data 322 for CP. In some embodiments, historical agentperformance attribution data 322 includes residual performances for allS in CP and static and dynamic performances for all defined factors.

In some embodiments, the calculated historical group 316 and agent 322performance attribution data is processed by the statistical analysisand scoring engine 324 using historical factor performance data 116 togenerate historical group 326 and agent 328 skill scoring data. In bothcases of group and agent performances, the inputs to statisticalanalysis and scoring engine 324 include a set of factor and residualmember performances realized by the group or agent for each observationperiod (e.g., all components of group active performance) as well as thedispersion measures of such sources of performance. The engine estimatesthe excess performance, if any, generated by these performance sourcesover the observation period and statistical significance of the excessperformance estimates (e.g., evidence of member selection and/or factortiming skills). For each component of performance attribution (staticfactor exposures, dynamic factor exposures, or residual memberexposures), the following process may be performed to determinestatistical significance. First, calculate the realized groupperformance due to static factor exposures, dynamic factor exposures, orresidual member exposures (also called herein group factor staticperformances, group factor dynamic performances, or member residualperformances, respectively). Second, calculate the expected performancedue to static factor exposures (also called herein group factor staticperformances, group passive performances, or group factor passiveperformances), dynamic factor exposures, or residual member exposures.In some embodiments, the expected performance due to static factorexposures is the group performance and the expected performance due todynamic factor exposures and residual member exposures is zero. Third,calculate the excess performance due to static factor, dynamic factor,or residual member exposures (also called herein group static factorperformances, group dynamic factor performances, or member residualperformances, respectively). Fourth, perform a statistical test T toreject the null hypothesis H0 that excess performance is zero. If H0 isrejected and estimated excess performance is greater than 0 withstatistical significance, then a corresponding agent is deemed to beskilled in dynamically timing factor(s) or selecting members generatingpositive residual performances. In some embodiments, a t-test isperformed to test the hypothesis H0. In some embodiments, tests that donot rely on the normal distribution of excess performances are performedto test the hypothesis H0. In some embodiments, Bayesian inference isused to test whether the region of practical equivalence for 0 excessperformance lies outside the highest density interval of the posteriordistribution for a particular confidence level (sometimes called a“reliability value”). In some embodiments, an alternative approach isused to evaluate factor timing skills. This approach focuses on a linearrelationship between initial group exposure to factor F and periodperformance for the factor F. If a significantly positive relationshipexists, the agent is deemed to be skilled in timing factor F byincreasing exposure to the factor F prior to periods where it generateshigher than average performance. In these embodiments, statisticalsignificance of the relationship can be tested by an F-test of theoverall fit and/or t-tests of individual parameters. In someembodiments, the excess performance data (group factor dynamicperformances and member residual performances) and the data associatedwith the statistical test such as the test statistics and confidencelevels are stored in data 326 or data 328.

For the test T, either ex-ante or ex-post dispersion metrics of excessperformance data may be used. The simplest form of this is theforecasted (for ex-ante) and realized (ex-post) standard deviation. Avariety of statistics or tests can be employed to produce more robustresults than those obtained from simple average and standard deviationmetrics, particularly for members with non-normal performancedistributions.

Historical group 326 and agent 328 skill scoring data generated by 324may also include any of the standard performance ratios and metrics suchas the reward-to-variability ratio and the Information Ratio generatedby exposure to individual factors and residual member risk. This datamay provide users with a familiar format for the presentation of moregranular data than group-level ratios.

For the purposes of presenting individual users with a simplifiedpicture of sources of active performance and their significance,beta-performance and beta-score are defined for the factor timing of thegroup and agent as a whole and each factor that groups and agents havebeen exposed to. Alpha-performance and alpha-score are similarly definedfor the member selection performance of the group and agent as a wholeand each member group, such as sector, that can be identified formembers included in historical groups and agents have been exposed to.These values are calculated by statistical analysis and scoring engine324. As used herein, beta-performance refers to a group factor timingperformance for a particular factor or factors. As used herein,beta-score refers to a confidence level that factor timing skills existfor a particular factor or factors. As used herein, alpha-performancerefers to a residual performance of a particular set of members. As usedherein, alpha-score refers to a confidence level that selection skillsexist for a particular set of members.

In some embodiments, the totality of the data produced by the system ismade available to Users 430 and third party applications 432. The dataavailable via reporting 422, query 424 and screening 426 user interfacesas well as APIs and web services 428 (externally available system data)includes historical group performance attribution data 316, agentperformance attribution data 322, group skill scoring data 326, agentskill scoring data 328 and miscellaneous group and agent data 410 thatmay be relevant for reports and queries. Reporting engine 412 processesthe externally available system data to generate group and agentperformance and evaluation reports, which can be stored in databases 418and 420 for caching and efficient distribution on duplicate requests.Reports may be generated to focus on factor and idiosyncraticperformances for specific sources such as individual factors,industries, geographies and size groupings. Reports may be generated toidentify and focus on areas of factor timing and member selection withsignificant positive or negative performances or evidence of skill.Reporting engine 412 may offer a comparison among sources of group oragent performance over history, and the evaluation of skill evolutionover history. Reporting engine 412, data query engine 414, and screeningengine 416 (the engines) may provide access to an objective orquantitative data on group or agent style (defined as factor exposureprofile) throughout group or agent history. The engines may also providean identification of group or agent style drift (change in factorexposure profile) or identify when group factor exposure profile ismaterially different from historical norms and outside the rangeconsistent with past performance.

In some embodiments, data query engine 414 provides access to theexternally available system data for the data query user interface 424and application programming interfaces (APIs) and web services 428.

In some embodiments, screening engine 416 exposes APIs that can be usedby the screening user interface 426 and external APIs and web services428 to filter for groups, agents, and any other externally availablesystem data matching specific criteria. In the most general case whereexternally available system data is stored in one or more SQL databases,the screening engine 416 allows execution of SELECT SQL queries spanningthe fields of such databases.

In some embodiments, screening user interface 426 provides users 430with the ability to build search queries (screens) spanning the fieldsof externally available system data using an intuitive graphicalinterface that does not require the knowledge of SQL or APIs. A sampleinterface showing a user-created screen is provided on FIG. 5. Using theinterface shown in FIG. 5, users have the ability to create, load andsave searches of groups and agents, including agent searches spanningdata from multiple groups associated with a particular agent. Theinterface allows users to select the entities to be searched for as wellas the performance metrics that matched entities must satisfy. In theexample shown in FIG. 5, a user chose to search for US coiled tubingdrillers using the performance of all drillers as the performancebenchmark for the evaluation. Multiple conditions can be added to thesearches. Each condition consists of a name of an externally availablesystem data field and a test that the data must satisfy. In the exampleshown in FIG. 5, seven conditions have been added including themagnitude and the statistical significance of the allocation or factortiming performances (Bperformance, Bscore), the magnitude and thestatistical significance of the selection performances (aperformance,ascore) and the magnitude of the selection performance for the 1,000horse power rigs sector (aperformance for the US coiled tubing drillerswith 1,000 horse power rating). The screen will identify agents withestimated skills in factor timing, overall driller selection andselection of US coiled tubing drillers. Additional conditions can beadded to further narrow the match set. In some embodiments, a user runsthe screen receiving a set of matching entities for the search, if any,and can further investigate the matches by accessing their reports viathe report UI 422 or accessing data via the data query UI 424.

In some embodiments, APIs and web services 428 provide access toexternal applications 432 to screening 416 and data query 414 engines aswell as reports databases 418 and 420.

Several modules and engines may be combined or may share code to achievea more efficient implementation. Common code from several modules can beextracted into shared libraries. For instance, web document downloaderand parser 212 may be combined or share code with the group data queryand conversion module 214.

Data can be stored in any of the common formats including delimited textfiles and SQL Tables. For optimal performance, all the user-accessibleperformance and scoring data should reside in storage that can beefficiently indexed and searched, such as a SQL database.

Data stored in multiple databases in the process flow charts may becombined in a single database for implementation and maintenance needs.For instance, some or all of the externally available data 316, 322,326, 328 and 410 may be combined in a single database.

Not all of member data sources 102 and 106 may be needed by animplementation of the system. For instance, a system with access tocomprehensive third party member data 106 may rely on third party memberdata 106 entirely.

Factor model specification 112 can consist of computer code capable ofestimating factor performances 116 and exposures 120, when executed byone or more processors of a computer system, or a high-level descriptiondirecting the sequence of modules and the flow of data among the modulesof a computer system to enable such estimation.

Multiple factor models may be specified and supported by the system andall of the values, statistics, and scores described can be calculatedfor a variety of models. Fundamental factor models define factors andmember exposures primarily via fundamental company properties.Statistical factor models define factors and member exposures primarilyvia statistical relationships among performance and other properties ofvarious baskets of members. The system can support any combination ofthese approaches.

When estimating factor performances and member factor exposures vialinear regression, multiple approaches can be used to limit the impactof outliers and overweight more recent observations. For instance,observations can be exponentially weighted to reduce the weighting ofolder data. Independent variables can also be weighted by the inverse ofthe regression residual to reduce the impact of outliers. Any othermethod of controlling the impact of outliers may be employed. Theweighting factors, exponential decay factors, and the historical periodover which regression data is collected can vary and may differ forvarious models. Models describing the performance of low-turnover groupswill generally use longer histories and be updated less frequently thanmodels describing the performance of high-turnover groups. In someembodiments, Bayesian linear regression (instead of the linearregression) is used for estimating member factor exposures.

Historical group member composition data 220 may be populated using asubset of the group data sources 202, 204, 206 and 208. For instance, anexemplary system may only process source data 206 using query andconversion module 214.

A variety of factor models may be used for performance attribution andevaluation. In some embodiments, sophisticated statistical modelsincluding principal components may be employed to identify the widevariety of systemic risks that may be taken. In some embodiments, asimpler fundamental or statistical model with a few sector and stylefactors may suffice. Users may have the ability to select the modelsused for performance analysis, reporting, and screening.

A variety of benchmarks may be used for performance reporting. Users mayhave the ability to select the benchmarks used for reporting, andscreening. However, the benchmark does not affect the evaluation offactor timing or member selection skills.

Optionally, group factor performance attribution engine 314 may generateand include in the historical group performance attribution data 316 theBrinson Fachler attribution or any other commonly used performanceattribution statistics. In some cases, arithmetic or geometricperformance decomposition (performance compounding) may be used.

Historical group performance attribution data 316 may include all of theindividual sub-components of a group or any values that can be used toestimate or derive this data. For instance, residual performances andgroup member exposures may be stored in place of residual performances.

Calculation of composite performance attribution for M over p(performance attribution data) may need to correct for theautoregressive aspects of member and factor performances when estimatingfactor and member volatility over longer periods via aggregation.

An alternative approach to developing a composite agent performance isto treat each group at period p as a set of independent factor andmember and selection decisions, preserving individual group attributiondata. In this approach, historical agent performance attribution data322 may contain a set of group factor performances and group residualperformances as well as their sub-components for each individual groupin each observation period p. The potential advantage of this is alarger dataset for processing by the statistical analysis and scoringengine 324.

When estimating excess performances and confidence levels for factor andresidual performances within statistical analysis and scoring engine324, non-normal distributions can be used to correct for fat tails.Additional tests can be performed to detect and correct forheteroskedasticity, autocorrelation, and multicollinearity.

The statistical analysis and scoring engine 324 can identify membersthat are not linear and where residual performances are not welldescribed by a normal distribution. In some embodiments, the engine 324estimates the probability of the various outcomes given the distributionof the underlying members and the resulting probability of the variousperformance outcomes, for instance using a binary tree. The engine canthen estimate the excess performance and confidence levels consistentwith the non-normal and non-linear nature of the instrument.

The statistical analysis and scoring engine 324 can define and calculatevalues in addition to beta-performance, beta-score, alpha-performanceand alpha-score that capture the excess performance of factor timing andmember selection and the statistical significance of such excessperformance to present users with a simplified picture of sources ofactive performance and their significance. These values can be given avariety of names for branding purposes or to improve their intuitivegrasp by the users.

Historical agent performance attribution and skill scoring may not beimplemented in development of a partial system providing performanceattribution and evaluation for groups only. In this case, database 322and 328 may not be implemented.

The user interfaces 422, 424 and 426 can be implemented for anddistributed on any platform including but not limited to: MicrosoftWindows, MacOS, or Unix desktops, Android or iOS mobile OperatingSystems, and Web Browsers using web programming technologies such asHTML5 and Flash.

The reporting user interface 422 can generate textual, table, orgraphical representation of all system data. For instance, performanceattribution can be displayed in table or graphical form. Specificfactor, group and agent performance attribution can be similarlydisplayed in graph form along the factor performance data. For example,scatter plots, distribution diagrams for various sources of activeperformances, and a number of additional approaches can be used toillustrate skill and/or positive bias in specific factor timing andmember selection performances, for instance cumulative performancecharts that illustrate the performances from timing specific factorsover the history.

Group performance attribution and skill reports can be generated by thesystem based on a number of settings or configuration parameters whichinclude the factor model used as well as other relevant variables. Thereports provide simple and intuitive summary of the factor timing andmember selection active performances in the form of the BPerformance andαPerformance values and the statistical significance of such activeperformances in the form of the βScore and αScore. Combined factortiming and member selection performances and their statisticalsignificance can also be provided in the form of αβPerformance andaBScore, respectively. The selection and factor timing activeperformances can be presented as percentiles of peer performances(αPercentile and βPercentile, respectively) to offer easy comparison tothe peers. Additional narrative can be generated to discuss the natureand the magnitude of any estimated skills in member selection and factortiming.

The user interfaces 422, 424 and 426 may be combined into one or moremodules providing aggregation of user interface features.

FIG. 5 provides a sample of a possible implementation of the screeningUI 426.

A variety of UI designs and implementations are possible. The commoncharacteristic of all search and screening interfaces is the ability toexecute queries spanning fields of the databases containing externallyavailable system information to generate a set of groups and agentsmeeting a given set of performance, skill evaluation and otherconditions.

Not all interface modules are necessary for system operation. Forinstance, an implementation using only reporting and screeninginterfaces implemented for Web browsers using HTML5 is possible. Anysubset of externally available system data being provided to users orapplications via a subset of user interfaces and engines may be used.

Group and agent historical factor exposure profile (style) may bedetermined from the historical exposure and performance decompositionreports. In some embodiments, changes in this profile or grouppositioning outside of historic norms, commonly referred to as “styledrift,” are explicitly pointed out in the reports and screens.

In some embodiments, a graphical representation of the range ofhistorical factor exposures is provided. In such embodiments, currentexposures (style) are plotted against the distribution to provide avisual illustration of style consistency.

In addition to a visual illustration of style consistency or drift, insome embodiments, a numeric score is computed and provided as anobjective measure of the difference (drift) of the current style awayfrom the historical norms for the group or agent.

FIG. 6 illustrates an exemplary user interface in accordance with someembodiments.

In FIG. 6, the user interface includes a display of a chart 610 and aconcurrent display of one or more user interface elements (e.g., 602,604, 606 and 608). In some embodiments, a respective user interfaceelement corresponds to a respective chart. For example, in someembodiments, user interface element 602 corresponds to a chart shown inFIG. 7; user interface element 604 corresponds to a chart shown in FIG.8; user interface element 606 corresponds to a chart shown in FIG. 9;and user interface element 608 corresponds to a chart shown in FIG. 10.In some other embodiments, the user interface includes more or feweruser interface elements. The correspondence between the user interfaceelements and charts may vary (e.g., user interface element 602 maycorrespond to a chart shown in FIG. 8 and user interface element 604 maycorrespond to a chart shown in FIG. 7, etc.).

In some embodiments, the respective user interface element, whenselected (e.g., by a touch input on a touch-sensitive display or a mouseclick), initiates a display of a corresponding chart in chart displayregion 610. As is evident from its name, chart display region 610 isconfigured to display a selected chart. For example, in someembodiments, user interface element 602, when selected, initiates adisplay of the chart shown in FIG. 7; user interface element 604, whenselected, initiates a display of the chart shown in FIG. 8; userinterface element 606, when selected, initiates a display of the chartshown in FIG. 9; and user interface element 608, when selected,initiates a display of the chart shown in FIG. 10. Thus, in someembodiments, while displaying a first chart, a selection of one of theone or more user interface elements is detected. In response todetecting the selection of one of the one or more user interfaceelements, the display of the first chart is replaced with a display of asecond chart that is distinct from the first chart.

FIG. 7 is a scatter plot that illustrates selected performanceindicators of groups in accordance with some embodiments.

The chart shown in FIG. 7 includes a plurality of data points. Arespective data point corresponds to a distinct group. The respectivedata point has an x-axis value that corresponds to an alpha performanceof the distinct group. The respective first data point has a y-axisvalue that corresponds to a beta performance of the distinct group.

In some embodiments, the alpha performance of the distinct groupcorresponds to an average group residual performance of the distinctgroup over one or more time periods. In some embodiments, the betaperformance of the distinct group corresponds to an average groupdynamic factor performance of the distinct group over the one or moretime periods.

As shown in FIG. 7, the respective data point is represented by acircle. In some embodiments, the size of the circle corresponds to anabsolute value of an alpha-beta performance of the distinct group. Insome embodiments, the color of the circle corresponds to a value of thealpha-beta performance (which may be positive or negative). In someembodiments, the alpha-beta performance corresponds to an average ofsums of historical group residual performances and historical groupdynamic factor performances for respective time periods of the one ormore time periods.

In some embodiments, the chart shown in FIG. 7 may be used to comparemember selection, factor timing, and aggregate active managementperformances across a group of groups or agents. In addition, the chartmay be used to obtain a quick visual comparison of group or agents'relative performance. In some cases, the chart may be used to identifygroups with the highest/lowest alpha/beta performances.

Alternatively, in some embodiments, the x-axis value may correspond toan alpha score of the distinct group, and the y-axis value maycorrespond to a beta score of the distinct group. In some embodiments,the size of the circle corresponds to an absolute value of an alpha-betascore of the distinct group and the color of the circle corresponds to avalue of the alpha-beta score. Such a chart would be similar to thechart shown in FIG. 7, but it focuses on reliability of performances andconfidence in the level of skills, rather than the value ofperformances.

In some embodiments, the alpha score of the distinct group correspondsto a confidence level of group residual performances of the distinctgroup over multiple time periods. In some embodiments, the beta score ofthe distinct group corresponds to a confidence level of group dynamicfactor performances of the distinct group over the multiple timeperiods. In some embodiments, the alpha-beta score corresponds to aconfidence level for sums of historical group residual performances andhistorical group dynamic factor performances for respective time periodsof the multiple time periods.

In some embodiments, a computer system detected a user input at alocation that corresponds to a particular data point. In response todetecting the user input, the computer system displays additionalinformation about a corresponding group. In some embodiments, theadditional information includes time series of one or more of alpha,beta, and alpha-beta performances. In some embodiments, the additionalinformation includes a chart illustrating distributions of one or moreof these performance series. In some embodiments, the additionalinformation includes graphs of one or more of these performance seriesover history. In some embodiments, multiple groups are concurrentlyselected (e.g., by concurrent or sequential user inputs). In response todetecting that multiple groups are selected, a computer systemconcurrently displays two or more of the above mentioned additionalinformation or overlay the additional information for two or moregroups.

FIG. 8 is a heat map that illustrates a selected performance indicatorof groups in accordance with some embodiments.

The chart shown in FIG. 8 illustrates a two-dimensional matrix of cells.One of a column of cells or a row of cells corresponds to a respectivetime period and the other of the column of cells and the row of cellscorresponds to a distinct group. For example, in FIG. 8, a horizontalstrip of cells corresponds to a time series of group residualperformances for a respective group.

In some embodiments, a color of a respective cell of the cellscorresponds to one of: the respective entry in the historical groupdynamic factor performance data, the respective entry in the historicalgroup residual performance data, a value that represents historicalgroup dynamic factor performance data for a predefined set of periods, avalue that represents cumulative historical group residual performancedata for a predefined set of periods, and a respective combined valuecorresponding to a sum of one or more respective values in thehistorical group dynamic factor performance data and one or morecorresponding values in the historical group residual performance data.In FIG. 8, a color of a respective cell corresponds to a magnitude ofthe alpha performance during a corresponding time period.

In some embodiments, a computer system detects a user input selecting arow of the cells (e.g., at a location that corresponds to a y-axis, suchas a name of a corresponding group). In response to detecting the userinput selecting the row of cells, the computer system displays a graphillustrating a time series of performances (e.g., residual performances)for the corresponding group over a plurality of time periods.

In some embodiments, the computer system detects a user input selectinga column of the cells (e.g., at a location that corresponds to anx-axis, such as a year or a time segment). In response to detecting theuser input selecting the column of cells, the computer system displays agraph illustrating performances (e.g., residual performances) of aplurality of groups for a corresponding time period.

In some embodiments, the computer system detects a user input selectinga particular cell on the chart, and displays one or more of theadditional information for a corresponding group for a correspondingtime period.

The chart shown in FIG. 8 illustrates may be used to identify periodsover which groups have generated significant positive and negative groupresidual performances. The chart in FIG. 8 may also be used to determinewhich historical regimes and events were associated with highvolatility, positive, and negative bias to such performances as well ascompare these patterns across a group of groups.

FIG. 9 is a correlation chart that illustrates similarity of groupsbased on a selected performance indicator in accordance with someembodiments.

The correlation chart shown in FIG. 9 illustrates a two-dimensionalmatrix that graphically represents correlation of performances of aplurality of groups.

In some embodiments, the correlation is determined based on one or moreof: the historical group dynamic factor performance data for respectivegroups and the historical group residual performance data for therespective groups. The chart in FIG. 9 is determined based on historicalgroup residual performances (i.e., alpha performances) of multiplegroups.

In FIG. 9, a respective ellipse represents a correlation value betweentwo groups. For example, an ellipse located at a second row from the topand a first column from the left represents a correlation value betweena first group named Group10 and a second group named Group26. In someembodiments, an ellipticity of the ellipse represents an absolute valueof the correlation value. In some embodiments, an orientation of theellipse represents whether the correlation value is positive ornegative. In some embodiments, a color of the ellipse represents a valueof the correlation value. In some embodiments, the respective ellipse isa circle (e.g., when the correlation is zero).

In some embodiments, a first portion of the two-dimensional matrixincludes a plurality of ellipses and a second portion of thetwo-dimensional matrix that does not overlap with the first portion ofthe two-dimensional matrix includes the plurality of numerical values.The plurality of ellipses graphically represents the plurality ofnumerical values. Each ellipse corresponds to a respective numericalvalue of the plurality of numerical values.

In some embodiments, a computer system detects a user input at alocation that corresponds to a respective ellipse. In response todetecting the user input at a location that corresponds to therespective ellipse, the computer system displays a scatter plot (forhistorical group dynamic factor performances for respective groups orhistorical group residual performances for the respective groups).

In some embodiments, the chart shown in FIG. 9 may be used to identifysimilarity of member selection strategies between any two groups over acertain time period (e.g., an entire history that corresponds toavailable data or a portion thereof). The chart may be also used toidentify similarity of member selection strategies between a group and agroup of groups for a certain time period. In some cases, the chart maybe used to identify groups or agents that are undifferentiated (i.e.,that have similar member selection strategies) to each other or betweena group and the group of groups. In other cases, the chart may be usedto identify groups and agents that are differentiated (i.e., that havedifferent member selection strategies).

FIG. 10 is a dendrogram that illustrates clustering of groups based on aselected performance indicator in accordance with some embodiments.

The dendrogram shown in FIG. 10 graphically illustrates clustering ofperformances of the plurality of groups. In FIG. 10, groups are groupedinto a tree structure in which each group is combined with another groupwith a closest correlation.

In some embodiments, the clustering is determined based on one or moreof: the historical group dynamic factor performance data for respectivegroups and the historical group residual performance data for therespective groups. For example, the dendrogram shown in FIG. 10 isdetermined based on the historical group residual performance data(i.e., alpha performances).

In some embodiments, a computer system detects a user input at alocation that corresponds to a node (or fork). In response to detectingthe user input at the location that corresponds to the node (or fork),the computer system displays a correlation scatter plot for a selectedperformance indicator for a first group and a second group, wherein thesecond group is a most closely correlated group to the first group withrespect to the selected performance indicator (i.e., alpha performance).In some embodiments, the computer system also displays a graph (e.g., ascatter plot) or table that illustrates historical correlation betweenthe first group and the second group with respect to the selectedperformance indicator.

In some embodiments, the dendrogram is used to identify correlationbetween any two groups. For example, the dendrogram may be used toidentify groups of groups with similar member selection strategies.Alternatively, or additionally, the dendrogram may be used to identifygroups of groups with similar factor timing strategies. In addition, thedendrogram may be used to identify groups of groups that havedifferentiated member selection strategies. In some cases, thedendrogram may be used to identify groups that provide the mostdifferentiation or diversification for a plurality of groups. In someother cases, the dendrogram may be used to identify groups or agentsthat pursue least or most unique member selection strategies within agroup of groups or agents.

In some embodiments, a user selection of a particular group (e.g., witha touch input on a touch-sensitive display or a mouse click) on one ofthe charts illustrated in FIGS. 7-10 initiates a display of a chart fora selected performance indicator. For example, in some embodiments, inresponse to selecting a row for a particular group from a chart shown inFIG. 8, a time series chart or a distribution chart for a selectedperformance indicator is displayed. In some embodiments, a chart for aselected performance indicator is displayed in response to a userselection user of a particular user interface element.

FIG. 11 is a block diagram illustrating exemplary distributed computersystem 1100 in accordance with some embodiments. In FIG. 11, system 1100includes one or more client computers 1102, communications network 1106,and performance quantification system 1108. Various embodiments ofperformance quantification system 108 implement the performancequantification methods described in this document.

Client computers 1102 can be any of a number of computing devices (e.g.,Internet kiosk, personal digital assistant, cell phone, gaming device,desktop computer, laptop computer, handheld computer, smart phones, orcombinations thereof) used to enable the activities described below.Client computer(s) 1102 is also referred to herein as client(s). Client1102 includes a graphical user interface (GUI) 1111. Client 1102 isconnected to performance quantification system 1108 via communicationsnetwork 1106. As described in more detail below, GUI 1111 is used todisplay group performance information. Performance quantification system1108 provides performance quantification services to a community ofusers (e.g., users of performance quantification system 1108) who accessperformance quantification system 1108 from clients 1102.

Performance quantification system 1108 includes one or more servers,such as server 1112, connected to communications network 1106.Optionally, the one or more servers are connected to communicationsnetwork 1106 via front end server 1122 (e.g., a server that conveys (andoptionally parses) inbound requests to the appropriate server of system1108, and that formats responses and/or other information being sent toclients in response to requests). Front end server 1122, if present, maybe a web server providing web based access to performance quantificationsystem 1108. Front end server 1122, if present, may also routecommunications to and from other destinations, such as remote (thirdparty) databases.

Performance quantification system 1108 includes historical factorperformance database 116 and historical group factor exposure database312. In some embodiments, performance quantification system 1108 alsoincludes or has access to one or more other databases, such ashistorical group performance attribution data 316, historical agentgroup association database 318, and database 1130 that stores otherperformance data. Server 1112 includes performance quantification module1124 and applications 128. Server 1112 communicates with databasesinternal to historical factor performance database 116, historical groupfactor exposure database 312, and in some embodiments, historical groupperformance attribution database 316, historical agent group associationdatabase 318, and database 1130 using a local area network, by internalcommunication busses, or by any other appropriate mechanism orcombination of mechanisms. In some embodiments, database 1130 includesother performance data such as historical group composition andperformance data 220 (FIG. 3).

Server 1112 communicates with clients 1102 via the front end server 1122(if present) and communication network(s) 1106. In some embodiments,communications network 1106 is the Internet. In other embodiments,communication network 1106 can be any local area network (LAN), widearea network (WAN), metropolitan area network, or a combination of suchnetworks. In some embodiments, server 1112 is a Web server that managesperformance quantification requests using appropriate communicationprotocols. Alternatively, if server 1112 is used within an intranet, itmay be an intranet server.

Applications 1128 include application programs used for managing aperformance quantification system. In some embodiments, applications1128 also include a user information processing module, where the userinformation processing module assists in accessing and updating a userinformation database (not shown). The user information database storesvarious information associated with the users of performancequantification system 1108, including user preferences, groups, andoptionally other information such as customized reports.

Performance quantification module 1124 retrieves and updates data storedin historical factor performance database 116 and historical groupfactor exposure database 312. Optionally, performance quantificationmodule 1124 also retrieves and updates data stored in historical groupperformance attribution database 316, historical agent group associationdatabase 318 and/or other performance database 1130.

Historical factor performance database 116 typically stores informationconcerning historical performances of various types of factors. Forexample, historical factor performance database 116 may includeinformation identifying that factor A had a performance of 8% in 1998and 3% in 1999, and factor B had a performance of 20% in 1998 and 15% in1999.

Historical group factor exposure database 312 typically storesinformation concerning historical factor exposure data for one or moregroups. For example, historical group factor exposure database 312 mayinclude information identifying that group A had 27% exposure to factorA, 52% exposure to factor B, and 21% exposure to factor C in 1998 andgroup B had 72% exposure to factor A, 13% exposure to factor B, and 15%exposure to factor C in 1998.

Historical group performance attribution database 316 typically includesinformation concerning historical performance attribution data for oneor more groups. For example, historical group performance attributiondatabase 316 may include information identifying that group A had aperformance of 13% in 1998 and 3% in 1999, and group B had a performanceof 20% in 1998 and 15% in 1999, all due to predefined factors. In someembodiments, historical group performance attribution database 316includes all the information for performances due to predefined factorsas well as residual performances due to individual group members.

Historical agent group association database 318 typically includesinformation concerning historical group association data for one or moreagents. For example, historical agent group association database 318 mayinclude information identifying that agent A is associated with group Afrom 1990 through 2000, group B from 2000 through 2010, and group C from2005 through 2013.

In essence, server 1112 is configured to manage certain aspects ofperformance quantification system 1108, including transmittingquantitative performance data to a respective client 1102.

In some embodiments, fewer and/or additional modules, functions ordatabases are included in performance quantification system 1108 andserver 1112. The modules shown in performance quantification system 1108and server 1112 represent functions performed in certain embodiments.

FIG. 12 is a block diagram illustrating computer system 1200 (e.g.,performance quantification system 1108) in accordance with someembodiments. Computer system 1200 typically includes one or moreprocessing units (CPUs) 1202, one or more network or othercommunications interfaces 1204, memory 1206, and one or morecommunication buses 1208 for interconnecting these components. In someembodiments, communication buses 1208 include circuitry (sometimescalled a chipset) that interconnects and controls communications betweensystem components. In some embodiments, computer system 1200 includes auser interface (not shown) (e.g., a user interface having a displaydevice, a touch screen, a touch pad, a keyboard, and a mouse or otherpointing device). In some other embodiments (e.g., when computer system1200 is implemented as a server), computer system 1200 is controlledfrom and accessed by various client systems.

Memory 1206 of computer system 1200 includes high-speed random accessmemory, such as DRAM, SRAM, DDR RAM or other random access solid statememory devices; and may include non-volatile memory, such as one or moremagnetic disk storage devices, optical disk storage devices, flashmemory devices, or other non-volatile solid state storage devices.Memory 1206 may optionally include one or more storage devices remotelylocated from the CPU(s) 1202. Memory 1206, or alternately thenon-volatile memory device(s) within memory 1206, comprises anon-transitory computer readable storage medium. In some embodiments,memory 1206 or the computer readable storage medium of memory 1206stores the following programs, modules and data structures, or a subsetthereof:

-   -   Operating System 1210 that includes procedures for handling        various basic system services and for performing hardware        dependent tasks;    -   Network Communication Module (or instructions) 1212 that is used        for connecting computer system 1200 to other computers (e.g.,        clients 1102) via one or more communications interfaces 1204 and        one or more communications networks 1106 (FIG. 11), such as the        Internet, other wide area networks, local area networks,        metropolitan area networks, and so on;    -   Performance Quantification Engine 1214 that receives performance        quantification requests from and provides responses to clients        1102; and    -   Presentation module 1220 that formats results from Performance        Quantification Engine 1214 for display at respective clients;        for example, presentation module 1220 may generate a web page or        XML document that includes performance information; in some        embodiments presentation module 1220 is executed by front end        server 1122 (FIG. 11), which comprises one of the servers        implementing performance quantification system; optionally        presentation module 1220 is a module of performance        quantification engine 1214.

In some embodiments, Performance Quantification Engine 1214 includes thefollowing programs, modules and data structures, or a subset or supersetthereof:

-   -   server 1112 for managing certain aspects of computer system 1200        including performance quantification module 1124, and        applications 1128, including performance quantification        application 1216 for performing the primary functions of a        performance quantification system; performance quantification        application 1216 includes one or more of alpha performance        module 1232, beta performance module 1234, alpha score module        1236, and beta score module 1238, and may optionally include        other modules;    -   Historical Factor Performance Database 116;    -   Historical Group Factor Exposure Database 312;    -   (Optional) Historical Group Performance Attribution Database        316;    -   (Optional) Historical Agent Group Association Database 318; and    -   (Optional) Other Performance Data 1130.

Each of the above identified modules and applications correspond to aset of instructions for performing one or more functions describedabove. These modules (i.e., sets of instructions) need not beimplemented as separate software programs, procedures or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various embodiments. In some embodiments, memory 1206 maystore a subset of the modules and data structures identified above.Furthermore, memory 1206 may store additional modules and datastructures not described above.

Notwithstanding the discrete blocks in FIGS. 11 and 12, these figuresare intended to be a functional description of some embodiments ratherthan a structural description of functional elements in the embodiments.One of ordinary skill in the art will recognize that an actualimplementation might have the functional elements grouped or split amongvarious components. In practice, and as recognized by those of ordinaryskill in the art, items shown separately could be combined and someitems could be separated. For example, in some embodiments, historicalfactor performance database 116 is part of or stored within server 112.In other embodiments, historical factor performance database 116 isimplemented using one or more servers whose primary function is to storeand process user information. In some embodiments, historical groupfactor exposure database 312 includes historical factor performancedatabase 116, or vice versa. Similarly, other performance database 1130can be implemented on one or more servers.

The actual number of servers used to implement performancequantification system 1108 and how features are allocated among themwill vary from one implementation to another, and may depend in part onthe amount of data traffic that the system must handle during peak usageperiods as well as during average usage periods, and may also depend onthe amount of data stored by the performance quantification system.Moreover, one or more of the blocks in FIG. 11 may be implemented on oneor more servers designed to provide the described functionality.Although the description herein refers to certain features implementedin client 1102 and certain features implemented in server 1112, theembodiments are not limited to such distinctions. For example, featuresdescribed herein as being part of server 1112 can be implemented inwhole or in part in client 1102, and vice versa.

FIGS. 13A-13G are flowcharts representing method 1300 of quantifyingperformance of a group in accordance with some embodiments. In someembodiments, method 1300 is performed by a computer system (e.g.,computer system 1200 in FIG. 12).

In some embodiments, the computer system receives (1302) a request toquantify factor timing performance associated with a respective group.In some embodiments, computer system 1200 (FIG. 12) receives a requestfrom client 1102 (FIG. 11). In some embodiments, computer system 1200receives a request from a user accessing computer system 1200. In someembodiments, computer system 1200 that receives the request is a client.In some embodiments, the request comprises a request to quantify factortiming performance associated with the respective group over a specificset of dates.

The computer system obtains (1304) historical performance data for aplurality of predefined factors. In some embodiments, the computersystem retrieves the historical performance data for the plurality ofpredefined factors from historical factor performance database 116 (FIG.12). For example, the computer system obtains historical performance(e.g., failure rates) of various classes of machines in a factory (e.g.,milling machine, lathe, welding machine, etc.). In some embodiments, thecomputer system receives the historical performance data for theplurality of predefined factors from another system or module (e.g.,factor performance estimator 114, FIG. 1). In some embodiments, thecomputer system receives the historical performance data for theplurality of predefined factors from a third party server. Thehistorical performance data for the plurality of predefined factorsincludes performance data for the plurality of predefined factors for aplurality of respective time periods. For example, the historicalperformance data for the plurality of predefined factors includes aperformance of a first factor for a first time period, a performance ofthe first factor for a second time period, a performance of the firstfactor for a third time period, a performance of a second factor for thefirst time period, a performance of the second factor for the secondtime period, and a performance of the second factor for the third timeperiod. A person having ordinary skill in the art would understand thatthe number of predefined factors (e.g., the first factor, the secondfactor, etc.) and the number of time periods (e.g., the first timeperiod, the second time period, the third time period, etc.) may vary.

The computer system obtains (1306) historical group factor exposure datafor the plurality of predefined factors for the respective group. Arespective entry in the historical group factor exposure data representsa relationship between a respective entry in historical performance dataof the respective group and a respective entry in historical performancedata of a respective factor of the plurality of predefined factors for arespective time period. In some embodiments, the historical group factorexposure data indicates what fraction of the machines at the factory isrepresented by each class of machines (e.g., 20% of the machines at thefactory are milling machines). Typically, the historical groupperformance data for the respective group includes performances of therespective group over a plurality of respective time periods. Forexample, the historical group performance data for the respective groupincludes a performance of the respective group for a first time period,a performance of the respective group for a second time period, aperformance of the respective group for a third time period, etc.Typically, the historical performance data of the respective factorincludes performances of the respective factor over the plurality ofrespective time periods. For example, the historical performance data ofthe respective factor includes a performance of the respective factorfor the first time period, a performance of the respective factor forthe second time period, a performance of the respective factor for thethird time period, etc. In some embodiments, the computer systemretrieves the historical group factor exposure data from historicalgroup factor exposure database 312 (FIG. 12). In some embodiments, thecomputer system retrieves the historical group factor exposure data fromanother system or module (e.g., group factor exposure estimator 310,FIG. 3). In some embodiments, the computer system retrieves thehistorical group factor exposure data from a third party server. In someembodiments, the computer system obtains the historical groupperformance data of the respective group from one or more of datasources shown in FIG. 2. In some embodiments, the computer systemdetermines the historical group performance of the respective group fora respective time period based on weights of members, that constitutethe respective group, in the respective group and performances of themembers. In some embodiments, the historical group factor exposure datais determined in accordance with historical performance data ofrespective members in the respective group, historical performance dataof the plurality of predefined factors, and weights of the respectivemembers in the respective group.

In some embodiments, a historical member factor exposure for therespective factor and the respective member of the respective groupcorresponds to a regression coefficient for a regression between thehistorical member performance data of the respective member of therespective group and the historical performance data of the respectivefactor. In some embodiments, the respective group corresponds to aweighted sum of the plurality of members, and a historical group factorexposure for the respective factor and the respective group correspondsto a sum of the exposures to the respective factor for the respectivemembers weighted by the group weight of the respective members.

The computer system generates (1308) historical group factor performancedata for the plurality of predefined factors for the respective group inaccordance with the historical performance data for the plurality ofpredefined factors and the historical group factor exposure data for theplurality of predefined factors for the respective group. In someembodiments, a group factor performance for a respective factor for therespective group for a respective time period corresponds to aperformance of the respective factor for the respective time periodmultiplied by a historical group factor exposure for the respectivefactor for the respective group for the respective time period. Forexample, a group factor performance for milling machines represents whatfraction of the overall failure rate for the factory is due to themilling machines in the factory.

The computer system stores (1309) the historical group factorperformance data for the plurality of predefined factors for therespective group.

The computer system generates (1310) historical group static factorperformance data for the plurality of predefined factors for therespective group in accordance with the historical performance data forthe plurality of predefined factors and one or more representativevalues of the historical group factor exposure data for the plurality ofpredefined factors for the respective group. In some embodiments, theone or more representative values of the historical group factorexposure data correspond to one or more average values of respectivesubsets of values in the historical group factor exposure data. In someembodiments, the one or more representative values of the historicalgroup factor exposure data for a respective factor for the respectivegroup includes an average of factor exposure values for the respectivefactor for the respective group over a plurality of time periods. Insome embodiments, a group static factor performance for the respectivefactor for the respective group for a respective time period correspondsto a performance of the respective factor for the respective time periodmultiplied by the average of factor exposure values for the respectivefor the respective group over the plurality of time periods.

The computer system stores (1311) the historical group static factorperformance data for the plurality of predefined factors for therespective group.

The computer system generates (1312, FIG. 13B) historical group dynamicfactor performance data for the plurality of predefined factors for therespective group in accordance with the historical group factorperformance data for the plurality of predefined factors for therespective group and the historical group static factor performance datafor the plurality of predefined factors for the respective group. Arespective entry in the historical group dynamic factor performance datarepresents the factor timing performance associated with the respectivegroup for a respective factor for a respective time period. In someembodiments, a group dynamic factor performance for a respective factorfor a respective time period corresponds to a difference between a groupfactor performance for the respective factor for the respective timeperiod and a historical group static factor performance for therespective factor for the respective time period.

The computer system stores (1313) the historical group dynamic factorperformance data for the plurality of predefined factors for therespective group.

The computer system provides (1314) one or more values that representthe historical group dynamic factor performance data. In someembodiments, the one or more values that represent the historical groupdynamic factor performance data include an average of values in thehistorical group dynamic factor performance data for one or morepredefined factors. In some embodiments, the computer system determinesand provides an average of group dynamic factor performances for arespective factor over a plurality of time periods. In some embodiments,the computer system determines and provides an average of group dynamicfactor performances for a plurality of factors for a respective timeperiod. In some embodiments, the computer system determines and providesan average of group dynamic factor performances for a plurality offactors over a plurality of time periods. In some embodiments, providingthe one or more values that represent the historical group dynamicfactor performance data includes graphically displaying the one or morevalues that represent the historical group dynamic factor performancedata (e.g., using one or more charts shown in FIGS. 7-10).

In some embodiments, the computer system, subsequent to generating thehistorical group dynamic factor performance data, determines (1316, FIG.13C) one or more values that indicate reliability (e.g., confidencelevel) of the historical group dynamic factor performance data; andprovides the one or more values that indicate reliability of thehistorical group dynamic factor performance data. In some embodiments,the one or more values that indicate the reliability of the historicalgroup dynamic factor performance data includes a confidence level for astatistical test comparing an average historical group dynamic factorperformance to 0.

In some embodiments, the computer system generates (1318) historicalgroup residual performance data for the respective group in accordancewith the historical group performance data for the respective group andthe historical group factor performance data for the plurality ofpredefined factors for the respective group. A respective entry in thehistorical group residual performance data represents member selectionperformance associated with the respective group. The computer systemprovides one or more values that represent the historical group residualperformance data. In some embodiments, the one or more values thatrepresent the historical group residual performance data include anaverage of values in the historical group residual performance data.Typically, the member selection performance associated with therespective group is not correlated with historical performance data forany one of the plurality of predefined factors.

In some embodiments, the computer system determines (1320) one or morevalues that indicate reliability (e.g., confidence level) of thehistorical group residual performance data; and provides one or morevalues that indicate reliability of the historical group residualperformance data. In some embodiments, the one or more values thatindicate the reliability of the historical group residual performancedata includes a confidence level for a statistical test comparing anaverage historical group residual performance to 0.

In some embodiments, the computer system determines (1322) one or morecombined values from the historical group dynamic factor performancedata and the historical group residual performance data, a respectivecombined value, of the one or more combined values, corresponding to asum of one or more respective values in the historical group dynamicfactor performance data and one or more corresponding values in thehistorical group residual performance data; and provides at least asubset of the one or more combined values.

In some embodiments, the computer system identifies (1324) a pluralityof groups associated with a respective agent. The plurality of groupsincludes the respective group. The computer system generates historicalgroup dynamic factor performance data for the plurality of predefinedfactors for particular groups of the plurality of groups; and providesone or more values that represent the historical group dynamic factorperformance data for the plurality of predefined factors for theparticular groups of the plurality of groups.

In some embodiments, the one or more values that represent thehistorical group dynamic factor performance data for the plurality ofpredefined factors for the particular groups of the plurality of groupsare provided (1326) in response to a request to quantify factor timingperformance of the respective agent.

In some embodiments, the computer system identifies (1328, FIG. 13C) aplurality of groups associated with a respective agent, wherein theplurality of groups includes the respective group. The computer systemgenerates historical group residual performance data for particulargroups in the plurality of groups; and provides one or more values thatrepresent the historical group residual performance data for theparticular groups of the plurality of groups.

In some embodiments, the one or more values that represent thehistorical group residual performance data for the particular groups ofthe plurality of groups are provided (1330) in response to a request toquantify member selection performance of the respective agent.

In some embodiments, a first group of the plurality of groups spans(1332) over a first time period and a second group of the plurality ofgroups spans over a second time period distinct from the first timeperiod. For example, start dates and/or end dates of the first group andthe second group may differ.

In some embodiments, the computer system identifies (1334) a respectiverange based on the historical group factor exposure data for arespective factor. In some embodiments, the respective range includesone or more of: a maximum, minimum, average, median, standard deviationof the historical group factor exposure data for a respective factor.The computer system determines whether a current group factor exposurefor the respective factor is outside of the respective range for therespective factor; and, in accordance with a determination that thecurrent group factor exposure for the respective factor is outside ofthe respective range for the respective factor, provides a notificationindicating that the current group factor exposure for the respectivefactor is outside the respective range for the respective factor. Insome embodiments, the notification is provided to a client device.Typically, that the current group factor exposure for the respectivefactor is outside the respective range for the respective factorindicates a style drift (of a group or its agent). In some embodiments,the method includes concurrently displaying the respective range and thecurrent group exposure for multiple factors.

In some embodiments, the historical group dynamic factor performancedata for a first factor represents a factor timing performanceassociated with the respective group for the first factor.

In some embodiments, the computer system displays (1336) a graphrepresenting entries in the historical group factor exposure data for asecond factor and entries in the historical performance data of thesecond factor, wherein the graph includes a plurality of data points, arespective data point having a first axis value that corresponds to arespective entry in the historical group factor exposure data for thesecond factor and a second axis value that corresponds to acorresponding entry in the historical performance data of the secondfactor.

In some embodiments, the computer system generates (1338) historicalgroup residual performance data for a subset of the respective group inaccordance with historical group performance data for the subset of therespective group and the historical group factor performance data forthe subset of the respective group. A respective entry in the historicalgroup residual performance data represents residual member selectionperformance associated with the subset of the respective group. In someembodiments, the respective group is divided based on one or moreselection criteria. For example, the subset may include membersbelonging to a particular sector, geographic location, type(electric-powered vs. gas-powered), or any other differentiatingcriteria. In some embodiments, the historical group residual performancedata for the subset of the respective group represents a memberselection performance associated with the respective group for somesectors or geographic locations (but not others). The historical groupresidual performance data for the subset of the respective group may beused to identify an agent with a reliable skill in selecting members ina specific country or sector.

In some embodiments, the computer system generates (1340, FIG. 13D)historical group residual performance data for a first group. The firstgroup includes members in the respective group and the members in thefirst group have same weights as each other. The computer systemprovides one or more values that represent the historical group residualperformance data for the first group.

In some embodiments, the method includes determining one or more valuesthat indicate reliability of the historical group residual performancedata for the first group; and providing one or more values that indicatereliability of the historical group residual performance data for thefirst group.

In some embodiments, the method includes determining a differencebetween a first value that represents the historical group residualperformance data for the respective group and a second value thatrepresents the historical group residual performance data for the firstgroup.

In some embodiments, the computer system generates (1341-1) historicalgroup residual performance data for a subset of the respective group inaccordance with historical group performance data for the subset of therespective group and the historical group factor performance data forthe subset of the respective group. A respective entry in the historicalgroup residual performance data represents residual member selectionperformance associated with the subset of the respective group. In someembodiments, the computer system generates historical group residualperformance data for a first group. The computer system generatesexpected residual performance data for the subset of the respectivegroup in accordance with one or more of the historical group residualperformance data for the respective group, the historical group residualperformance data for the subset of the respective group, a difference ofthe historical group residual performance data for the subset of therespective group and historical group residual performance data for afirst group, and current group factor exposure data for the subset ofthe respective group. The first group includes members in the respectivegroup and the members in the first group have same weights as eachother. The computer system stores the expected residual performance datafor the subset of the respective group. In some embodiments, ahistorical expected residual performance for a subset of the respectivegroup is a weighted sum of a historical group residual performance forthe respective group, a historical group residual performance for thesubset of the respective group, a difference between the historicalgroup residual performance data for the subset of the respective groupand the historical group residual performance for the first group, acurrent group factor exposure for the subset of the respective group,and a multiple of the difference between the historical group residualperformance data for the subset of the respective group and thehistorical group residual performance for the first group and a currentgroup factor exposure for the subset of the respective group.

In some embodiments, the computer system repeats (1341-2) generatingexpected residual performance data for distinct subsets of therespective group. The computer system generates expected residualperformance data for the respective group in accordance with theexpected residual performance data for the distinct subsets of therespective group. The computer system stores the expected residualperformance data for the respective group. In some embodiments, anexpected residual performance for the respective group is a weighted sumof a historical group residual performance for the respective group,historical group residual performance for the subsets of the respectivegroup, differences between the historical group residual performancedata for the distinct subsets of the respective group and the historicalgroup residual performance for corresponding first groups, and currentgroup factor exposure for the subsets of the respective group.

In some embodiments, the historical group factor exposure data isdetermined (1342, FIG. 13F) in accordance with historical member factorexposure data and historical group member exposure data. In someembodiments, the computer system determines the historical group factorexposure data in accordance with the historical member factor exposuredata and the historical group member exposure data (e.g., as weightedsums of historical member factor exposures, weighted by historical groupmember exposures).

In some embodiments, the computer system displays (1344) a first chartselected from a set of charts (e.g., a user interface illustrated inFIG. 6 with region 610 including a chart shown in FIG. 7).

In some embodiments, the set of charts includes a first chart thatincludes a plurality of first data points (e.g., FIG. 7). A respectivefirst data point corresponds to a distinct group or a set of groups. Therespective first data point has a first axis value that represents oneof: the historical group dynamic factor performance data for thedistinct group or the set of groups and a value that indicatesreliability of the historical group dynamic factor performance data. Therespective first data point has a second axis value that represents oneof: the historical group residual performance data for the distinctgroup or the set of groups and a value that indicates reliability of thehistorical group residual performance data.

In some embodiments, the set of charts includes a second chart thatillustrates a two-dimensional matrix of cells (e.g., FIG. 8). One of acolumn of cells or a row of cells corresponds to a respective timeperiod and the other of the column of cells and the row of cellscorresponds to a distinct group or a set of groups. A respective cell ofthe cells corresponds to one of: the respective entry in the historicalgroup dynamic factor performance data, the respective entry in thehistorical group residual performance data, a value that representscumulative historical group dynamic factor performance data for apredefined set of periods, a value that represents cumulative historicalgroup residual performance data for a predefined set of periods, and arespective combined value corresponding to a sum of one or morerespective values in the historical group dynamic factor performancedata and one or more corresponding values in the historical groupresidual performance data.

In some embodiments, the set of charts includes a correlation chart thatillustrates a two-dimensional matrix that graphically representscorrelation of performances of a plurality of groups (e.g., FIG. 9). Thecorrelation is determined based on one or more of: the historical groupdynamic factor performance data for respective groups and the historicalgroup residual performance data for the respective groups.

In some embodiments, the set of charts includes a dendrogram thatgraphically illustrates clustering of performances of the plurality ofgroups (e.g., FIG. 10). The clustering is determined based on one ormore of: the historical group dynamic factor performance data forrespective groups and the historical group residual performance data forthe respective groups.

In some embodiments, the computer system concurrently displays one ormore user interface elements with the first chart (e.g., user interfaceelements 602, 604, 606, and 608 in FIG. 6). While displaying the firstchart and the one or more user interface elements (e.g., FIG. 6), thecomputer system detects a selection of one of the one or more userinterface elements (e.g., on user interface element 604). In response todetecting the selection of one of the one or more user interfaceelements, the computer system replaces the display of the first chartwith a display of a second chart (e.g., FIG. 7), wherein the secondchart is selected from the set of charts and the second chart isdistinct from the first chart.

In some embodiments, the computer system generates (1346) expecteddynamic factor performance data for a respective factor in accordancewith the historical group dynamic factor performance data and currentgroup factor exposure data for the respective group for the respectivefactor. The computer system stores the expected dynamic factorperformance data for the respective factor. In some embodiments, anexpected dynamic factor performance for the respective factor is aweighted sum of a historical group dynamic factor performance for therespective group and a historical group dynamic factor performance forthe respective factor.

In some embodiments, the computer system repeats (1348) generatingexpected dynamic factor performance data for distinct factors of therespective group. The computer system generates expected dynamic factorperformance data for the respective group in accordance with theexpected dynamic factor performance data for the distinct factors. Thecomputer system stores the expected aggregate dynamic factor performancedata for the respective group. In some embodiments, an expected dynamicfactor performance for the respective group is a weighted sum of ahistorical group dynamic factor performance for the respective group andhistorical group dynamic factor performance values for the distinctfactors.

FIG. 14 is a conceptual diagram illustrating generation of a highperforming group in accordance with some embodiments. In FIG. 14, firstgroup 1400 includes a plurality of elements 1402, 1404, 1406, 1408,1410, 1412, 1414, 1416, 1418, and 1420. Second group 1430 includes aplurality of elements 1432, 1434, 1436, 1438, 1440, 1442, 1444, 1446,1448, and 1450. In some embodiments, the plurality of elements in firstgroup 1400 and second group 1430 are individual members. For example,each element is a machine in a factory. In some embodiments, theplurality of elements in first group 1400 and second group 1430 aregroups, such that first group 1400 is a first set of groups, and secondgroup 1430 is a second set of groups. For example, each element is agroup of machines in a factory.

In the example shown in FIG. 14, high alpha-scores are associated withhigh group performance (e.g., high historical group residualperformance). Each element in first group 1400 has a respectivealpha-score: element 1402 has an alpha-score of 99; element 1404 has analpha-score of 94; element 1406 has an alpha-score of 15; element 1408has an alpha-score of 37; element 1410 has an alpha-score of 4; element1412 has an alpha-score of 58; element 1414 has an alpha-score of 82;element 1416 has an alpha-score of 64; element 1418 has an alpha-scoreof 18; and element 1420 has an alpha-score of 96.

Similarly, each element in second group 1430 has a respectivealpha-score: element 1432 has an alpha-score of 28; element 1434 has analpha-score of 94; element 1436 has an alpha-score of 67; element 1438has an alpha-score of 90; element 1440 has an alpha-score of 51; element1442 has an alpha-score of 47; element 1444 has an alpha-score of 87;element 1446 has an alpha-score of 33; element 1448 has an alpha-scoreof 89; and element 1450 has an alpha-score of 9.

A third group 1460 is formed using elements in group 1400 and group 1430that have high alpha-scores. As shown in FIG. 14, third group 1460includes element 1402, element 1404, element 1420, element 1434, element1438, and element 1448. In some embodiments, such as in the exampleshown in FIG. 14, a predetermined number of highest scoring elements ineach of group 1400 and group 1430 are selected for inclusion in group1460. For example, the three elements with the highest alpha-scores ingroup 1400 (e.g., element 1402 with alpha-score 99, element 1420 withalpha-score 96, and element 1404 with alpha-score 94) are selected forinclusion in group 1460, and the three elements with the highestalpha-scores in group 1430 (e.g., element 1434 with alpha-score 94,element 1438 with alpha-score 90, and element 1448 with alpha-score 89)are selected for inclusion in group 1460. In some embodiments wherefirst group 1400 includes a plurality of members and second group 1430includes a plurality of members, third group 1460 is a group of members.In some embodiments where first group 1400 is a first set of groups andsecond group 1430 is a second set of groups, third group 1460 is a setof groups that includes respective subsets of groups from first group1400 and second group 1430.

In some embodiments, elements having alpha-scores (or, more generally,respective reliability values) that are above a predefined threshold areselected for inclusion in group 1460. For example, if the predefinedthreshold alpha-score is 90, then group 1460 would include element 1402with alpha-score 99, element 1404 with alpha-score 94, element 1420 withalpha-score 96, element 1434 with alpha-score 94, and element 1438 withalpha-score 90; but, unlike the example shown in FIG. 14, group 1460would not include element 1448 with alpha-score of only 89, which isless than 90.

In some embodiments, third group 1460 is formed by specifying desiredalpha-performance criteria and/or desired alpha-score criteria usingscreening user interface 426 (FIG. 4), an example of which is presentedin FIG. 5, so as to generate a set of groups meeting the specifiedcriteria.

FIGS. 15A-15B are graphs illustrating example distributions ofreliability values in accordance with some embodiments. In someembodiments, the reliability values indicate reliability of dataassociated with a respective measure of performance, such as reliabilityof historical group residual performance data for a respective group. Inanother example, the reliability values indicate reliability ofhistorical group dynamic factor performance data. In yet anotherexample, the reliability values indicate reliability of autocorrelationdata (e.g., reliability of sets of autocorrelation values) for arespective measure of performance.

FIG. 15A illustrates an even distribution of reliability values,associated with a respective measure of performance, across a range ofreliability values r₁ through r_(N). In some embodiments, as describedherein, the reliability values are obtained using a statistical test totest a null hypothesis. In some embodiments, an even distribution ofreliability values associated with a respective measure of performanceindicates that the reliability values that reject the null hypothesisare attributable to random chance given the number of reliability valuesobserved, and, accordingly, that the null hypothesis cannot be rejectedon the basis of a particular reliability value that otherwise (inisolation) would have been used to reject the null hypothesis. In someembodiments, therefore, an even distribution of reliability valuesassociated with a respective measure of performance indicates thatperformance—high, low, or otherwise—is also attributable to randomchance. For example, as described herein, a statistical test T isperformed to reject a null hypothesis H0 that excess performance (e.g.,residual performance) is below zero. In some cases, the determination ofa single reliability value R using the statistical test T indicates thatH0 is rejected and estimated excess performance is greater than 0 withstatistical significance, such that a corresponding agent may be deemedto be skilled in dynamically timing factor(s) or selecting membersgenerating positive residual performances. However, if the determinationof multiple reliability values indicates an even distribution ofreliability values, then the single reliability value R is notsufficient to reject H0. In other words, although the single reliabilityvalue R indicates that estimated excess performance is greater than 0with statistical significance, the occurrence of the reliability valueR, and the corresponding positive estimated excess performance, isattributable to random chance. In some cases, an even distribution ofreliability values is associated with a valid null hypothesis H0 despiteevidence to the contrary from a single test of H0. In some embodiments,other analysis of the aggregate properties set of all reliability valuesis used to determine whether H0 can be rejected for a specificreliability value.

FIG. 15B illustrates an uneven distribution of reliability values,associated with a respective measure of performance, across the range ofreliability values r₁ through r_(N). In particular, FIG. 15B illustratesa higher occurrence of reliability values at the upper and lower ends ofthe range of reliability values. In some embodiments, where thereliability values are obtained using a one-tailed (also called“one-sided”) statistical test for testing a null hypothesis, an unevendistribution of reliability values as shown in FIG. 15B indicates thatthe reliability values are unlikely to occur due to random chance. Forexample, as described herein, a statistical test T is performed toreject a null hypothesis H0 that excess performance (e.g., residualperformance) is below zero. In some cases, the determination of a singlereliability value R using the statistical test T indicates that H0 isrejected and estimated excess performance is greater than 0 withstatistical significance, such that a corresponding agent may be deemedto be skilled in dynamically timing factor(s) or selecting membersgenerating positive residual performances. If the statistical test T isone-tailed, a resulting distribution of reliability values (e.g., forall agents) as shown in FIG. 15B indicates that values near one end ofthe range of reliability values (e.g., near r₀) support rejecting H0.Optionally, in some embodiments, values near one end of the range ofreliability values (e.g., near r₀) support rejecting H0 in one manner(e.g., due to excess performance in one direction, such as excesspositive performance), and values near the opposite end of the range ofreliability values (e.g., near r_(N)) support rejecting H0 in adifferent manner (e.g., due to excess performance in the oppositedirection, such as excess negative performance).

FIGS. 16A-16B are graphs illustrating example autocorrelation valuesassociated with performance of a respective group in accordance withsome embodiments. In some embodiments, a respective autocorrelationvalue indicates a degree of correlation between one or more valuesassociated with a respective measure of performance during a first timeperiod and one or more values associated with the respective measure ofperformance during a time period that occurs after the first timeperiod. As an example, and for ease of explanation, FIGS. 16A-16B aredescribed with reference to historical group residual performance for arespective group.

In FIG. 16A, autocorrelation value a₁ indicates a positive correlationbetween group residual performance during a first time period and groupresidual performance during a second time period that occurs after atime T₁ from the first time period (e.g., one year after the first timeperiod). Autocorrelation value a₂ indicates a positive correlationbetween group residual performance during the first time period andgroup residual performance during a third time period that occurs aftera time T₂ from the first time period (e.g., two years after the firsttime period). Autocorrelation value a₃ indicates a positive correlationbetween group residual performance during the first time period andgroup residual performance during a third time period that occurs aftera time T₃ from the first time period (e.g., three years after the firsttime period), and so on and so forth through autocorrelation valuea_(N-1) for a time period that occurs after a time T_(N-1) from thefirst time period (e.g., N−1 years after the first time period).Autocorrelation value a_(N), however, indicates a negative correlationbetween group residual performance during the first time period andgroup residual performance during an Nth time period that occurs after atime T_(N) from the first time period (e.g., N years after the firsttime period). In some cases, the high occurrence and/or consistency ofpositive autocorrelation values in the graph of FIG. 16A is interpretedas indicating that past group residual performance is predictive ofsubsequent group residual performance. Further, in some cases, the graphof FIG. 16A is interpreted as indicating that selection skills wereexhibited (e.g., by an agent) in selecting one or more members of therespective group for which the autocorrelation values in FIG. 16A weregenerated.

In FIG. 16B, autocorrelation value b₁ indicates a positive correlationbetween group residual performance during a first time period and groupresidual performance during a second time period that occurs after atime T₁ from the first time period (e.g., one year after the first timeperiod). Autocorrelation value b₂ indicates a positive, but lesser,correlation between group residual performance during the first timeperiod and group residual performance during a third time period thatoccurs after a time T₂ from the first time period (e.g., two years afterthe first time period). Autocorrelation value b₃ indicates a negativecorrelation between group residual performance during the first timeperiod and group residual performance during a third time period thatoccurs after a time T₃ from the first time period (e.g., three yearsafter the first time period). Autocorrelation value b₄ indicates an evenmore negative correlation between group residual performance during thefirst time period and group residual performance during a fourth timeperiod that occurs after a time T₄ from the first time period (e.g.,four years after the first time period). Autocorrelation values throughb_(N-1) and b_(N) in FIG. 16B are neither consistently positive norconsistently negative. In some cases, the low occurrence and/orinconsistency of positive autocorrelation values in the graph of FIG.16B is interpreted as indicating that past group residual performance isat best weakly predictive of subsequent group residual performance.Accordingly, the graph of FIG. 16B cannot be interpreted as indicatingthat selection skills were exhibited (e.g., by an agent) in selectingone or more members of the respective group for which theautocorrelation values in FIG. 16B were generated, because theautocorrelation values neither include a high occurrence of positivevalues nor are consistently positive for more than two time periods T₁and T₂.

FIGS. 17A-17G are flowcharts representing method 1700 of quantifyingperformance of a group in accordance with some embodiments. In someembodiments, method 1700 is performed by a computer system (e.g.,computer system 1200 in FIG. 12).

In some embodiments, the computer system receives (1702) a request toquantify group residual performance associated with a first group. Asdescribed herein with reference to FIG. 14, in some embodiments thefirst group is group 1400, while in some embodiments the first group isa respective element such as element 1402. In some embodiments, “groupresidual performance” refers to the residual performance that is due tothe selection of members into the group. In some embodiments, computersystem 1200 receives a request from a user accessing computer system1200. In some embodiments, computer system 1200 that receives therequest is a client. In some embodiments, the request comprises arequest to quantify group residual performance associated with the firstgroup over a specific set of dates.

The computer system obtains (1704) historical group performance data forthe first group. In some embodiments, the computer system retrieves thehistorical group performance data for the first group from historicalgroup composition and performance database 220 (FIG. 3). For example,the computer system obtains historical performance data of a group ofmachines in a factory. In some embodiments, the computer system receivesthe historical group performance data for the first group from anothersystem or module (e.g., from web crawler, document downloader, andparser 212, group data query and conversion engine 214, and/orhistorical group management user interface 218, FIG. 2). In someembodiments, the computer system receives the historical groupperformance data from a third party server. The historical groupperformance data includes performance data for the first group for aplurality of respective time periods. For example, the historical groupperformance data for the first group includes performance of the firstgroup (e.g., performances of the members of the first group) for a firsttime period, performance of the first group for a second time period,performance of the first group for a third time period, and so on. Aperson having ordinary skill in the art would understand that the numberof time periods (e.g., the first time period, the second time period,the third time period, etc.) may vary.

The computer system obtains (1706) historical group factor exposure datafor a plurality of predefined factors for the first group (e.g., asdescribed herein with reference to operation 1306 of method 1300).

The computer system obtains (1708) historical factor performance datafor the plurality of predefined factors for the first group (e.g., asdescribed herein with reference to operation 1308 of method 1300).

The computer system generates (1710) historical group residualperformance data for the first group in accordance with the historicalgroup performance data for the first group, the historical group factorexposure data for the plurality of predefined factors for the firstgroup, and the historical factor performance data for the plurality ofpredefined factors for the first group. A respective entry in thehistorical group residual performance data represents member selectionperformance associated with the first group (e.g., for a particular timeperiod). For example, a respective entry in the historical groupresidual performance data for a group of machines represents memberselection performance for the group of machines during a respective timeperiod. In some embodiments, the historical group residual performancedata for the first group is generated by group factor performanceattribution engine 314 (FIG. 3). In some embodiments, the historicalgroup residual performance data for the first group includes alphaperformance data for the first group. In some embodiments, alphaperformance data for the first group is generated by alpha performancemodule 1232 (FIG. 12).

In some embodiments, the computer system stores (1712) the historicalgroup residual performance data for the first group (e.g., in historicalgroup performance attribution database 316, FIG. 11, or in alphaperformance module 1232, FIG. 12).

In some embodiments, the computer system provides (1714) one or morevalues that represent the historical group residual performance data. Insome embodiments, the provided one or more values include respectivevalues that represent the historical group residual performance forrespective time periods. In some embodiments, the provided one or morevalues include respective alpha values that represent the historicalgroup residual performance (e.g., alpha performance) for respective timeperiods.

In some embodiments, one or more reliability values that indicatereliability of the historical group residual performance data aredetermined, and the determined one or more reliability values thatindicate reliability of the historical group residual performance dataare provided. In some embodiments, reliability values described hereinare generated by statistical analysis and scoring engine 324 (FIG. 3).In some embodiments, reliability values described herein are stored inhistorical group skill scoring database 326 (FIG. 3).

For example, a determined reliability value is an alpha-score, where thealpha-score indicates the statistical significance of the historicalgroup residual performance data for a respective group (e.g., the firstgroup). In some embodiments, an alpha-score indicates a confidence levelthat selection skills exist for a respective group (e.g., the firstgroup). That is, an alpha-score indicates a confidence level that skillwas exhibited in selecting one or more members of the respective group.In some embodiments, the alpha-score is a value between 0% and 100%. Insome embodiments, alpha-scores are generated by and/or stored in alphascore module 1236 (FIG. 12).

In another example, a determined reliability value is a p-value, wherethe p-value is obtained using a one-tailed t-test of the historicalgroup residual performance data for a respective group. In someembodiments, the p-value is related to the alpha-score. In someembodiments, the p-value is a value between 0 and 1, and is equal to1−alpha-score/100.

In yet another example, the determined reliability value is asignal-to-noise ratio. In some embodiments, a respective reliabilityvalue (e.g., alpha-score) for a respective group indicates a likelihoodthat the historical group residual performance data for the respectivegroup is attributable to member selection skill (e.g., skill of anagent, responsible for selecting the one or more members of therespective group, in selecting the one or more members) rather than tochance.

In some embodiments, the first group includes (1716) a plurality ofmembers, the plurality of members including at least one member of afirst type and at least one member of a second type that is differentfrom the first type. For example, as described herein with reference toFIG. 14, group 1400 includes a plurality of elements, which may beindividual members. In some embodiments, the first group includes theplurality of members without regard to member type. In some embodiments,the historical group residual performance data is generated for thefirst group without regard to member type. A member can be any object orentity for which it would be desirable to quantify performance. A membertype can any classification of a member. For example, the first groupcan be a group of machines that includes at least one machine of a firsttype (e.g., a milling machine) and at least one machine of a second type(e.g., a welding machine). In this example, historical group residualperformance data generated for the group of machines (as a whole) isagnostic to machine type. For example, where the elements in group 1400are machines, group 1400 includes member machines 1402, 1404, 1406,1408, 1410, 1412, 1414, 1416, 1418, and 1420 without regard to whether aparticular machine is of a particular type (e.g., milling machine,welding machine, lathe, etc.).

In some embodiments, the first group includes (1718) a plurality ofmembers, and all of the plurality of members in the first group are of afirst type. For example, the first group includes only machines of afirst type (e.g., only milling machines). Referring again to FIG. 14, insome embodiments, group 1400 includes only milling machines. In thisexample, historical group residual performance data can be quantified ona type-by-type basis. That is, historical group residual performancedata generated for the group of machines represents historical groupresidual performance of milling machines. In some embodiments, the firstgroup is formed from a larger group that includes a plurality of membersof the first type, and one or more members of one or more typesdifferent from the first type. For example, a larger group includes aplurality of milling machines, and at least one or more lathes orwelding machines, and the first group is formed from the larger groupsuch that the first group includes only milling machines (e.g., all ofthe milling machines from the larger group). Referring again to FIG. 14,in some embodiments, group 1400 is a subgroup of a larger group thatincludes other types of machines besides milling machines (e.g., alarger group that also includes (sub)group 1430, where group 1430includes member machines 1432, 1434, etc., that are a different type ofmachine).

In some embodiments, the first group is (1720) a respective group in aplurality of groups. In some embodiments, the plurality of groupsincludes a second group that includes a plurality of members. And, insome embodiments, all of the plurality of members in the second groupare of a second type that is different from the first type. In someembodiments, each group in the plurality of groups includes members of adifferent type. In some embodiments, the historical group residualperformance data for the first group does not include historical groupresidual performance data for other groups in the plurality of groups.In some embodiments, the historical group residual performance data forthe first group represents (only) historical residual performance ofmembers of the first type (e.g., and does not represent historicalresidual performance of member of any type other than the first type).For example, the first group (e.g., element 1402, FIG. 14) includes onlymilling machines and the second group (e.g., group 1430, FIG. 14)includes only welding machines (e.g., elements 1432, 1434, 1436, 1438,1440, 1442, 1444, 1446, 1448, 1450, FIG. 14). In this example,historical group residual performance data generated for the first grouprepresents historical group residual performance of (only) millingmachines, and historical group residual performance data generated forthe second group represents historical group residual performance of(only) welding machines.

In some embodiments, the first group is (1722) a respective group in afirst set of groups, the first set of groups including a plurality ofgroups, and each respective group in the first set of groups including aplurality of members. Referring again to FIG. 14, for example, the firstgroup is element 1402 in a first set of groups 1400, the first set ofgroups 1400 including a plurality of groups 1402, 1404, 1406, etc. Insome embodiments, for each respective group in the first set other thanthe first group (e.g., for groups 1404, 1406, etc.), the computer systemobtains: historical group performance data for the respective group;historical group factor exposure data for a plurality of predefinedfactors for the respective group; and historical factor performance datafor the plurality of predefined factors for the respective group. Insome embodiments, for each respective group in the first set other thanthe first group, the computer system generates respective historicalgroup performance data for the respective group in accordance with thehistorical group performance data for the respective group, thehistorical group factor exposure data for the plurality of predefinedfactors for the respective group, and the historical factor performancedata for the plurality of predefined factors for the respective group.In some embodiments, a respective entry in the respective historicalgroup residual performance data for the respective group representsmember selection performance associated with the respective group (e.g.,for a particular time period). In some embodiments, the respectivehistorical group residual performance data for the respective group inthe first set is stored at the computer system (e.g., in historicalgroup performance attribution database 316, FIG. 11, or in alphaperformance module 1232, FIG. 12). In some embodiments, a respective setof one or more values that represent the respective historical groupresidual performance data for the respective group is provided.

In some embodiments, the computer system identifies (1724) a subset ofgroups in the first set based at least in part on the respectivehistorical group residual performance data for each respective group inthe first set. In some embodiments, the subset of groups consists ofless than all of the groups in the first set. In some embodiments, thecomputer system generates a second set of groups that consists of thesubset of groups, and provides one or more values that represent thesecond set of groups (e.g., including providing identifiers and weightsfor the groups in the second set, and/or providing identifiers andweights for the members in (the groups in) the second set). In someembodiments, the one or more values that represent the second set areprovided in response to receiving a request to generate a highperforming set of groups. In some embodiments, the second set of groupsis provided. In some embodiments, generating the second set of groupsincludes applying a weight to each of the groups in the subset of thefirst set, where the second set includes the weighted groups. Forexample, groups 1402, 1404, and 1420 in set 1400 are identified based onthe respective alpha-scores of those groups. In some embodiments, asecond set of groups, consisting of groups 1402, 1404, and 1420, isgenerated. In some embodiments, the second set of groups consists ofgroup 1402 with a first weight w₁ applied, group 1404 with a secondweight w₂ applied, and group 1420 with a third weight w₃ applied.

In some embodiments, identifying the subset of groups in the first setis (1726) performed without regard to whether a respective groupincludes members of a particular type. For example, identifying a subsetof groups of machines is performed without regard to the type(s) ofmachines that make up each group of machines (e.g., without regard towhether any particular machine is a milling machine, a lathe, a weldingmachine, or any other type of machine). In this example, the identifiedsubset of groups of machines include highest performing (groups of)machines without regard to machine type. In some embodiments, the subsetof groups includes at least one member of a first type and at least onemember of a second type that is different from the first type. In someembodiments, the subset of groups includes at least one group thatincludes at least one member of the first type, and at least one group(e.g., the same group or a different from the group that includes the atleast one member of the first type) that includes at least one member ofthe second type. Referring again to FIG. 14, for example, the groups inset 1400 may include different types of machines.

In some embodiments, all of the members in the first set of groups are(1728) of a first type. Accordingly, all of the members in the secondset of groups are of the first type. Referring again to FIG. 14, forexample, if all of the members of the groups in set 1400 are millingmachines, then all of the members in the groups in the second set ofgroups consisting of groups 1402, 1404, and 1420 are also millingmachines. In some embodiments, the one or more values that represent thesecond set of groups are provided in response to receiving a request togenerate a high performing group for a particular member type. In someembodiments, the first set of groups are formed from a larger group thatincludes a plurality of members of the first type, and one or moremembers of one or more types different from the first type. For example,a larger group includes a plurality of milling machines, and at leastone or more lathes or welding machines, and the first set of groups(e.g., set 1400, FIG. 14) is formed from the larger group such that thefirst set of groups includes only (groups of) milling machines (e.g.,every milling machine from the larger group is placed into a respectivegroup in the type-specific first set of groups).

In some embodiments, all of the members in the first set of groups are(1730) of a first type. For example, all of the members in set 1400(FIG. 14) are milling machines. In some embodiments, a third set ofgroups includes a plurality of groups, each respective group in thethird set of groups including a plurality of members. In someembodiments, all of the members in the third set of groups are of asecond type that is different from the first type. For example, set 1430(FIG. 14) includes a plurality of groups 1432, 1434, 1436, etc., eachincluding a plurality of members. In some embodiments, all of themembers in set 1430 are welding machines. In some embodiments, for eachrespective group in the third set (e.g., for each group 1432, 1434,1436, etc.), the computer system: obtains historical group performancedata for the respective group; obtains historical group factor exposuredata for a plurality of predefined factors for the respective group; andobtains historical factor performance data for the plurality ofpredefined factors for the respective group. In some embodiments, foreach respective group in the third set, the computer system generatesrespective historical group residual performance data for the respectivegroup (in the third set) in accordance with the historical groupperformance data for the respective group, the historical group factorexposure data for the plurality of predefined factors for the respectivegroup, and the historical factor performance data for the plurality ofpredefined factors for the respective group. In some embodiments, arespective entry in the historical group residual performance data forthe respective group represents member selection performance associatedwith the respective group (in the third set) (e.g., for a particulartime period). In some embodiments, the respective historical groupresidual performance data for the respective group in the third set isstored at the computer system (e.g., in historical group performanceattribution database 316, FIG. 11, or in alpha performance module 1232,FIG. 12). In some embodiments, a respective set of one or more valuesthat represent the respective historical group residual performance datafor the respective group in the third set is provided.

In some embodiments, the computer system identifies a subset of groups(e.g., less than all of the groups) in the first set based at least inpart on the respective historical group residual performance data foreach respective group in the first set (e.g., as described herein withreference to operation 1724). In some embodiments, the computer systemidentifies a subset of groups (e.g., less than all of the groups) in thethird set based at least in part on the respective historical groupresidual performance data for each respective group in the third set.For example, groups 1434, 1438, and 1448 in set 1430 (FIG. 14) areidentified based on the respective alpha-scores of those groups. In someembodiments, the computer system generates a fourth set of groups (e.g.,set 1460, FIG. 14) that includes the identified subset of groups in thefirst set (e.g., groups 1402, 1404, and 1420 from set 1400) and theidentified subset of groups in the third set (e.g., groups 1434, 1438,and 1448 from set 1430). In some embodiments, the computer systemprovides one or more values that represent the fourth set of groups(e.g., including providing identifiers and weights for the groups in thefourth set, and/or providing identifiers and weights for the members in(the groups in) the fourth set).

In some embodiments, the one or more values that represent the fourthset are provided in response to receiving a request to generate a highperforming group for a plurality of member types (e.g., including thefirst type and the second type). In some embodiments, the fourth set ofgroups is provided. In some embodiments, generating the fourth setincludes applying a weight to each group in the identified subset ofgroups in the first set and in the identified subset of groups in thesecond set, where the fourth set includes the weighted groups. In someembodiments, historical group residual performance data is generated forany number of sets of groups, each set including members of a particulartype that is different from the member type of other sets. In someembodiments, respective historical group residual performance data isgenerated for each respective group in each respective set of groups. Insome embodiments, the predetermined number of highest performing groupsin each set is identified. In some embodiments, a new set of groups isgenerated that includes (e.g., aggregates) the predetermined number ofhighest performing groups across all groups in the sets of groups.

In some embodiments, a respective entry in the respective historicalgroup residual performance data for a respective subset represents(1732) member selection performance associated with the respectivesubset during a respective time period in a predefined set of timeperiods. In some embodiments, identifying the subset of groups in arespective set includes identifying a predetermined number of groupshaving the highest respective entries in the respective historical groupresidual performance data during a particular time period in thepredefined set of time periods. In some embodiments, the predeterminednumber of groups is specified by a user via a user interface (e.g.,screening user interface 426, FIG. 4).

In some embodiments, a respective entry in the respective historicalgroup residual performance data for a respective subset represents(1734) member selection performance associated with the respectivesubset during a respective time period in a predefined set of timeperiods. In some embodiments, identifying the subset of groups in arespective set includes identifying groups having respective entries inthe respective historical group residual performance data that satisfy apredefined threshold during a particular time period in the predefinedset of time periods. In some embodiments, the predefined threshold forhistorical group residual performance is specified by a user via a userinterface (e.g., screening user interface 426, FIG. 4). In someembodiments, entries with high values are associated with highperformance. In some embodiments, groups having entries that exceed afirst predefined threshold (e.g., a high threshold) are identified, soas to identify high performing groups. In some embodiments, groupshaving entries that are below a second predefined threshold (e.g., a lowthreshold) are identified, so as to identify low performing groups. Insome embodiments, the low threshold is a negative value. For example,identifying high performing groups of machines enables those groups ofmachines to be selected and used to perform tasks, whereas identifyinglow performing groups of machines can be useful in selecting whichgroups of machines on which to perform maintenance or repairs.

In some embodiments, the computer system determines (1736), for eachrespective group, a respective reliability value that indicatesreliability of the respective historical group residual performance dataof the respective subset over a predefined set of time periods. In someembodiments, the determined reliability values are alpha-scores, whereeach alpha-score indicates the statistical significance of therespective historical group residual performance data for the respectivesubset. In some embodiments, each alpha-score indicates a confidencelevel that selection skills exist for the respective subset (e.g., skillin selecting the one or more members of the respective subset). In someembodiments, the determined reliability values are p-values, where eachp-value is obtained using a one-tailed t-test of the respectivehistorical group residual performance data for the respective subset. Insome embodiments, the determined reliability values are signal-to-noiseratios, such as ratios quantifying a relationship between residualperformance and a standard deviation of the residual performance (e.g.,Sharpe ratios). In some embodiments, a respective reliability value(e.g., alpha-score) for a respective subset indicates a likelihood thatthe respective historical group residual performance data for therespective subset is attributable to member selection skill (e.g., skillof an agent, responsible for selecting the one or more members of therespective subset, in selecting the one or more members).

In some embodiments, identifying the subset of groups in a respectiveset includes identifying a predetermined number of groups having thehighest respective reliability values. In some embodiments, thepredetermined number of groups is specified by a user via a userinterface (e.g., screening user interface 426, FIG. 4). In someembodiments, a high alpha-score is associated with high performancereliability, or high consistency of positive performance. In someembodiments, a predetermined number of groups N in a set of groups areidentified, where the N groups have the N highest alpha-scores of allthe groups in the set of groups. For example, as described herein withreference to FIG. 14, the three elements from set 1400 with the highestalpha-scores are identified. In some embodiments, a low p-value isassociated with high performance reliability, or high consistency ofpositive performance. In some embodiments, a predetermined number ofgroups N in a set of groups are identified, where the N groups have theN lowest p-values of all the groups in the set of groups. In someembodiments, a high Sharpe ratio is associated with high performancereliability, or high consistency of positive performance. In someembodiments, a predetermined number of groups N in a set of groups areidentified, where the N groups have the N highest Sharpe ratios of allthe groups in the set of groups.

In some embodiments, the computer system determines (1738), for eachrespective group, a respective reliability value that indicatesreliability of the respective historical group residual performance dataof the respective subset over a predefined set of time periods. In someembodiments, identifying the subset of groups in a respective setincludes identifying groups having respective reliability values thatsatisfy a predefined threshold. In some embodiments, the predefinedthreshold is specified by a user via a user interface (e.g., screeninguser interface 426, FIG. 4). In some embodiments, groups havingreliability values (e.g., alpha-scores) that exceed a first predefinedthreshold (e.g., a high alpha-score threshold) are identified; in somesuch embodiments, the identified groups are high performing groups. Forexample, as described herein with reference to FIG. 14, elements havingalpha-scores that are 90 or above are identified. In some embodiments,groups having reliability values (e.g., p-values) that are below asecond predefined threshold (e.g., a low p-value threshold) areidentified; in some such embodiments, the identified groups are highperforming groups. In some embodiments, groups having reliability values(e.g., p-values) that are above a third predefined threshold (e.g., ahigh p-value threshold) are identified; in some such embodiments, theidentified groups are low performing groups. In some embodiments,identifying groups based on whether their respective reliability valuessatisfy a predefined threshold may result in any number of identifiedgroups (e.g., and is not limited to a predefined number of groups).

In some embodiments, each respective time period in the predefined setof time periods spans (1740) one month. In some embodiments, eachrespective time period in the predefined set of time periods spans oneday. In some embodiments where a respective time period in thepredefined set of time periods spans one month instead of one day, theamount of data processing is reduced without significant loss ofinformation (e.g., relative to embodiments where a respective timeperiod spans one day). For example, generating historical group residualperformance data from obtained data on a monthly basis instead of on adaily basis (e.g., where the data itself may be obtained on a dailybasis) reduces the amount of data processing and/or transmissionrequired (e.g., by roughly 30 times), thereby enabling the computersystem to operate more quickly and efficiently, without incurring acorresponding reduction in the efficacy of the methods described hereinin identifying high performing groups.

In some embodiments, each respective time period in the predefined setof time periods spans a longer interval than the intervals for which thedata is obtained (e.g., collected). For example, where data is obtainedfor each day, historical group residual performance data is generatedfor each week. In another example, where data is obtained for each day,historical group residual performance data is generated for each month.One of ordinary skill will appreciate that other pairings of shorterdata-collection intervals with longer residual performance intervals canbe used, such as obtaining data every time a respective value in thedata changes (e.g., several times in one second) and determininghistorical group residual performance data for each minute or hour.

In some embodiments, the computer system identifies (1742) adistribution of values associated with the reliability values (e.g., asdescribed herein with reference to FIGS. 15A-15B). In some embodiments,the computer system determines: a first number of reliability valuesthat are within a first range within the distribution of values (e.g., arange including high-confidence reliability values). In someembodiments, the first range includes values in the lowest 5% of thedistribution (e.g., where the reliability values are p-values). In someembodiments, the first range includes values in the highest 5% of thedistribution (e.g., where the reliability values are alpha-scores).

In some embodiments, the computer system determines: whether the firstnumber of reliability values exceeds a (first) threshold number ofreliability values by at least a predefined amount. In some embodiments,the threshold number of reliability values is determined based on thetotal number of the reliability values and on the first range. In someembodiments, the threshold number of reliability values corresponds tothe expected number of reliability values in the first range that wouldbe observed by chance. In some embodiments, the first range represents afraction (or percentage) of the distribution of values, and thethreshold number of reliability values is the total number ofreliability values multiplied by the fraction (or percentage). Forexample, if the first range includes values in 5% of the distribution(e.g., the lowest 5% or the highest 5%), the threshold number ofreliability values is 5% of the total number of reliability values. Insome embodiments, the threshold number of reliability values is apredefined multiple (e.g., two, three, four, etc.) of the total numberof reliability values multiplied by the fraction (of the first rangewith respect to the distribution). For example, if the first rangeincludes values in 5% of the distribution, the threshold number ofreliability values is 10%, 15%, 20%, etc. of the total number ofreliability values.

In some embodiments, identifying the subset of groups in a respectiveset is further based on a determination that the first number ofreliability values exceeds the threshold number of reliability values byat least the predefined amount.

In some embodiments, a second number of reliability values that arewithin a second range within the distribution of values is determined(e.g., a range including low-confidence reliability values). In someembodiments, the second range includes values in the highest 5% of thedistribution (e.g., where the reliability values are p-values). In someembodiments, the second range includes values in the lowest 5% of thedistribution (e.g., where the reliability values are alpha-scores). Insome embodiments, the computer system determines whether the secondnumber of reliability values exceeds a second threshold number ofreliability values by at least the predefined amount. In someembodiments, the second threshold number of reliability values isdetermined based on the total number of the reliability values and onthe second range. In some embodiments, the second threshold number ofreliability values corresponds to the expected number of reliabilityvalues in the second range that would be observed by chance. In someembodiments, the second range represents a second fraction (orpercentage) of the distribution of values, and the second thresholdnumber of reliability values is the total number of reliability valuesmultiplied by the second fraction (or percentage). In some embodiments,identifying the subset of groups in a respective set is further based ona determination that the second number of reliability values exceeds thesecond threshold number of reliability values by at least the predefinedamount. In some embodiments, ranges other than the lowest and thehighest 5% may be used for the first and second ranges, such as thelowest and the highest 10%, or the lowest and the highest thirds.

In some embodiments, an even distribution of the reliability valuesindicates that member selection performance consistency of the groups israndomly distributed, which in turn indicates that member selectionperformance consistency is attributable to chance (e.g., as describedherein with reference to FIG. 15A). In some embodiments, an unevendistribution of reliability values over the range of values indicatesthat member selection performance consistency of the groups is notrandomly distributed. For example, where the reliability values arep-values obtained from a one-tailed t-test of historical group residualperformance values, in some cases a high occurrence of p-values below afirst threshold level of significance (e.g., in a lowest range, such asthe lowest 5%, of the possible range of p-values) relative to theexpected occurrence of p-values below the first threshold level ofsignificance for the total number of reliability values distributed overthe range of values is observed. This indicates that member selectionperformance consistency (with respect to positive performance) occurswith statistical significance, at a greater rate than would be seen bychance. Optionally, in conjunction with the high occurrence of p-valuesbelow the first threshold level of significance, a high occurrence ofp-values above a second threshold level of significance (e.g., in ahighest range, such as the highest 5%, of the possible range ofp-values) relative to the expected occurrence of p-values above thesecond threshold level of significance is observed. This indicates thatmember selection performance consistency (with respect to both positiveperformance and negative performance) occurs with statisticalsignificance, at a greater rate than would be seen by chance. Forexample, FIG. 15B illustrates a high occurrence of reliability values(such as p-values) at the upper end of the range of reliability valuesand at the lower end of the range of reliability values, with a loweroccurrence of reliability values in the middle of the range ofreliability values.

In some cases, the range of values associated with p-values is the rangeof values from 0 to 1. In some such cases, the total number of p-valuesthat are between 0 and 0.05 (the lowest 5% of the range of values) isthe first number. Optionally, in some such cases, the total number ofp-values that are between 0.95 and 1 (the highest 5% of the range ofvalues) is the second number. In some such cases, a determination thatthe total number of p-values between 0 and 0.05 exceeds the total numberof p-values between 0 and 0.05 that would be observed by chance by atleast a predefined amount (e.g., by 100% or more) indicates that memberselection performance consistency of the groups is attributable to(positive) member selection skill (e.g., skill producing positiveperformance) with statistical significance. In some such cases, adetermination that the total number of p-values between 0.95 and 1exceeds the total number of p-values between 0.95 and 1 that would beobserved by chance by at least a predefined amount (e.g., by 100% ormore) indicates that member selection performance consistency isattributable to (negative) member selection skill (e.g., skill producingnegative performance) with statistical significance. One of ordinaryskill in the art will appreciate that this analysis can be performedusing any other metric of reliability (e.g., type of reliability value),such as alpha-scores. The observed distribution of the reliabilityvalues can be compared to the distribution that would have been observedby chance, so as to determine whether the reliability values aredistributed non-randomly.

In some embodiments, subsets of multiple sets of groups, each set ofgroups with a different member type, are selected in accordance with adetermination that reliability values for each set of groups indicatethat member selection performance consistency associated with therespective member type of the respective set of groups is not randomlydistributed. For example, a first subset of a first set of groups ofmachines of a first type is selected based at least in part on adetermination that residual performance consistency for machines of thefirst type is not randomly distributed, and occurs with statisticalsignificance. A second subset of a second set of groups of machines of asecond type is selected based at least in part on a determination thatresidual performance consistency for machines of the second type is notrandomly distributed, and occurs with statistical significance.

In some embodiments, where performance (e.g., historical groupperformance data) of a respective set of groups is monitored repeatedly(e.g., continuously or periodically) over time, in accordance with adetermination that reliability values for the respective set of groupsis randomly distributed (e.g., neither the total occurrence ofreliability values in the lowest 5% of the range of values, nor, in someembodiments, the total occurrence of reliability values in the highest5% of the range of values exceeds the respective total occurrences ofreliability values in these ranges that would be expected by chance),the performance of the respective set of groups ceases to be monitored.Ceasing to monitor performance of groups for which residual performanceconsistency is randomly distributed (and thus not indicative ofstatistically significant member selection performance) reduces theamount of data processing required by the computer system, which enablesthe computer system to operate more quickly and efficiently.

In some embodiments, the respective historical group residualperformance data for a respective group includes (1744): respectivehistorical group residual performance data for the respective group overa first set of time periods; and respective historical group residualperformance data for the respective group over one or more subsequentsets of time periods. In some embodiments, the one or more subsequentsets of time periods occur later in time than the first set of timeperiods. In some embodiments, the computer system determines, for eachrespective group, one or more respective autocorrelation valuesindicating a degree of autocorrelation between the respective historicalgroup residual performance data for the respective group during thefirst set of time periods and the respective historical group residualperformance data for the respective group during the one or moresubsequent sets of time periods (e.g., as described herein withreference to FIGS. 16A-16B). In some embodiments, identifying the subsetof groups in a respective set is further based at least in part on adetermination that autocorrelation values for the subset of groupssatisfy predefined autocorrelation criteria.

In some embodiments, the predefined autocorrelation criteria aresatisfied when the autocorrelation values for the groups in the subsetof groups satisfy a respective threshold (e.g., greater than zero). Insome embodiments, the predefined autocorrelation criteria are satisfiedwhen autocorrelation values for the groups in the subset of groupssatisfy the respective threshold (e.g., are greater than zero) for atleast a predefined number of subsequent sets of time periods (e.g.,autocorrelation values for a respective group are positive with respectto all of at least three, four, or five subsequent time periods). Forexample, autocorrelation values in FIG. 16A are positive with respect tomore than four, and as many as N−1, subsequent time periods, which insome cases satisfies predefined autocorrelation criteria. In acontrasting example, autocorrelation values in FIG. 16B are positivewith respect to only two subsequent time periods, which in some casesfails to satisfy predefined autocorrelation criteria. In someembodiments, the predefined autocorrelation criteria are satisfied whenthe significance values for autocorrelation values for the groups in thesubset of groups satisfy a respective threshold (e.g., their p-valuesare below 5%, also expressed as 0.05). For example, repeated occurrenceof positive autocorrelation values in FIG. 16A corresponds to a p-valueof less than 0.05. In a contrasting example, autocorrelation values thatare neither consistently positive nor consistently negative correspondto a p-value of greater than 0.05. One of ordinary skill in the art willrecognize that a one-tailed t-test, or another statistical testappropriate for the particular distribution of autocorrelation values,can be used to determine the significance values for the autocorrelationvalues.

In some embodiments, subsets of multiple sets of groups, each set ofgroups with a different member type, are selected in accordance with adetermination that autocorrelation values for each set of groupsindicate that past residual performance of groups associated with therespective member type of the respective set of groups is predictive of(e.g., correlated with) subsequent residual performance of groups of thesame respective member type, which in turn is interpreted as beingpredictive of, or likely to be predictive of, future residualperformance of groups of the same respective member type. For example, afirst subset of a first set of groups of machines of a first type isselected based at least in part on a determination that past residualperformance of machines of the first type is predictive of subsequentresidual performance of machines of the first type (e.g., by exhibitingpositive autocorrelation between residual performance during differentperiods in time). A second subset of a second set of groups of machinesof a second type is selected based at least in part on a determinationthat past residual performance of machines of the second type ispredictive of subsequent residual performance of machines of the firsttype (e.g., by exhibiting positive autocorrelation between residualperformance during different periods in time (e.g., the same periods intime as the first characteristic)).

In some embodiments, where performance (e.g., historical groupperformance data) of a respective group is monitored repeatedly (e.g.,continuously or periodically) over time, in accordance with adetermination that autocorrelation values for the respective group doesnot satisfy the predefined autocorrelation criteria, the performance ofthe respective group ceases to be monitored. Ceasing to monitorperformance of groups for which past performance during a first periodin time is not predictive of subsequent performance during subsequentperiods in time (and thus likely not predictive future performanceduring future periods in time) reduces the amount of data processingrequired by the computer system, which enables the computer system tooperate more quickly and efficiently.

In some embodiments, the computer system obtains (1746), for each memberin the first group, historical use data for the respective member (e.g.,starting periods and ending periods of use or exercise of the respectivemember). In some embodiments, the historical use data for a respectivemember includes starting periods and ending periods of use, selection,activation, or exercise of the respective member. In some embodiments,the time between a respective starting period and ending period pairingindicates time during which the respective member is a member of arespective group (e.g., time during which the respective member isselected for the respective group).

In some embodiments, the computer system generates historical groupmember use data for the first group in accordance with the respectivehistorical use timing data for each respective member in the firstgroup. In some embodiments, the historical group member use data for thefirst group includes starting periods and ending periods of use for allmembers and for all time periods represented by the historical use data.In some embodiments, obtaining the historical group performance data forthe first group includes: obtaining historical member performance datafor the first group; and generating the historical group performancedata for the first group in accordance with the historical group memberuse data for the first group and the historical member performance datafor the first group. In some embodiments, obtaining the historical groupfactor exposure data for the plurality of predefined factors for thefirst group includes: obtaining historical member factor exposure datafor the first group; and generating the historical group factor exposuredata for the plurality of predefined factors for the first group inaccordance with the historical group member use data for the first groupand the historical member factor exposure data for the first group.

In some embodiments, obtaining the historical group factor exposure datafor the plurality of predefined factors for the first group includes(1748): obtaining (estimated) initial historical group factor exposuredata for the plurality of predefined factors for the first group; andgenerating an aggregate historical group factor exposure value for theplurality of predefined factors for a group including all possiblemembers (e.g., with respective weights applied), based on the initialhistorical group factor exposure data for the plurality of predefinedfactors for the first group. In some embodiments, obtaining thehistorical group factor exposure data for the plurality of predefinedfactors for the first group further includes: determining an offsetbetween the aggregate historical group factor exposure value for theplurality of predefined factors for the group including all possiblemembers and an expected aggregate historical group factor exposurevalue. In some embodiments, the determined offset is a positive value.In some embodiments, the determined offset is a negative value. In someembodiments, obtaining the historical group factor exposure data for theplurality of predefined factors for the first group further includesgenerating the historical group factor exposure data for the pluralityof predefined factors for the first group based on the initialhistorical group factor exposure data for the plurality of predefinedfactors for the first group and the determined offset.

For example, a group including all possible members includes theuniverse of all machines (e.g., all machines in operation). In someembodiments, the expected aggregate historical group factor exposurevalue for all machines is expected to be 1.0. In some cases, however,the aggregate historical group factor exposure value, determined basedthe (estimated) initial historical group factor exposure data for theplurality of predefined factors, is less than the expected 1.0. Forexample, the determined aggregate historical group factor exposure valueis 0.9 instead. In some embodiments, an offset (e.g., here, 0.1) isdetermined between the expected and determined aggregate historicalgroup factor exposure values. In some embodiments, the (estimated)initial historical group factor exposure data for the plurality ofpredefined factors is adjusted (e.g., here, increased) by the determinedoffset to generate (adjusted) historical group factor exposure data forthe plurality of predefined factors for the first group, such that theaggregate historical group factor exposure value, when determined fromthe adjusted historical group factor exposure data, is 1.0 as expected.

In some embodiments, the first group includes a plurality of subgroups.In some embodiments, obtaining the historical group performance data forthe first group includes (1750) obtaining historical group performancedata for at least a first subgroup in the first group and a secondsubgroup in the first group. In some embodiments, the first subgroupcorresponds to a first time period. In some embodiments, the secondsubgroup corresponds to a second time period distinct from the firsttime period. In some embodiments, the provided historical group residualperformance data for the first group represents historical groupresidual performance data for the second subgroup in the first group. Insome embodiments, the first group is associated with a particular agent(e.g., plurality of groups in the first group are all associated withthe agent). In some embodiments, the provided historical group residualperformance data for the first group represents the performance of theagent associated with the first group. In some embodiments, where thesecond subgroup is associated with the agent, and the second time periodcorresponding to the second subgroup includes insufficient data fordetermining historical group residual performance data for the secondsubgroup alone, historical group residual performance of a plurality ofgroups, including one or more other subgroups associated with the sameagent (and optionally including the second subgroup) is used torepresent, or in some embodiments predict, the historical group residualperformance of the second subgroup.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, to therebyenable others skilled in the art to best utilize the invention andvarious embodiments with various modifications as are suited to theparticular use contemplated.

For example, in some embodiments, a method for quantifying performanceassociated with a group includes receiving, at a computer system withone or more processors and memory, a request to quantify memberselection performance of a respective group (e.g., group residualperformance of the respective group). The method includes obtaining, atthe computer system, historical performance data for a plurality ofpredefined factors; and obtaining, at the computer system, historicalgroup factor exposure data for the plurality of predefined factors for arespective group, wherein the historical group factor exposure dataincludes information identifying a respective relationship betweenhistorical group performance data of the respective group and historicalperformance data of a respective factor of the plurality of predefinedfactors. The method also includes generating, at the computer system,historical group factor performance data for the plurality of predefinedfactors for the respective group in accordance with the historicalperformance data for the plurality of predefined factors and thehistorical group factor exposure data for the plurality of predefinedfactors for the respective group. The method further includesgenerating, at the computer system, historical group residualperformance data for the respective group in accordance with historicalgroup performance data for the respective group and the historical groupfactor performance data for the plurality of predefined factors for therespective group, wherein a respective entry in the historical groupresidual performance data represents member selection performance of therespective group; and providing one or more values that represent thehistorical group residual performance data.

In some embodiments, a method for providing performance data formultiple groups includes displaying a first chart selected from a set ofcharts.

In some embodiments, the set of charts includes a scatter plot (e.g.,FIG. 7) that includes a plurality of first data points, a respectivefirst data point corresponding to a distinct group or a set of groups,the respective first data point having a first axis value thatrepresents one of: historical group dynamic factor performance data forthe distinct group or the set of groups and a value that indicatesreliability of the historical group dynamic factor performance data anda second axis value that represents one of: historical group residualperformance data for the distinct group or the set of groups and a valuethat indicates reliability of the historical group residual performancedata.

In some embodiments, the set of charts includes a heat map (e.g., FIG.8) that illustrates, for a plurality of groups and a plurality of timeperiods, one of: a respective entry in the historical group dynamicfactor performance data, a respective entry in the historical groupresidual performance data, a value that represents cumulative historicalgroup dynamic factor performance data for a predefined set of timeperiods, a value that represents cumulative historical group residualperformance data for a predefined set of time periods, and a respectivecombined value corresponding to a sum of one or more respective valuesin the historical group dynamic factor performance data and one or morecorresponding values in the historical group residual performance data.

In some embodiments, the set of charts includes a correlation chart(e.g., FIG. 9) that graphically represents correlation of performancesof a plurality of groups, wherein the correlation is determined based onone or more of: the historical group dynamic factor performance data forrespective groups, and the historical group residual performance datafor the respective groups.

In some embodiments, the set of charts includes a dendrogram (e.g., FIG.10) that graphically illustrates clustering of performances of theplurality of groups, wherein the clustering is determined based on oneor more of: the historical group dynamic factor performance data forrespective groups, and the historical group residual performance datafor the respective groups.

The method includes concurrently displaying one or more user interfaceelements with the first chart; while displaying the first chart and theone or more user interface elements, detecting a selection of one of theone or more user interface elements; and, in response to detecting theselection of the one of the one or more user interface elements,replacing the display of the first chart with a display of a secondchart. The second chart is selected from the set of charts and thesecond chart is distinct from the first chart.

In some embodiments, the methods and system described herein can be usedwith other entities, such as mechanical procedures, chemical processes,investment portfolios, and medical procedures, etc.

Definitions

As used herein, the following terms are defined as shown below.

Factor—a variable used to describe variability among observed,correlated variables. In some cases, a factor is an unobserved variable.For example, a factor can be failure rates of a class of machines at afactory, and the factor can be used to describe a failure rate of allmachines at the factory.

Factor analysis—a statistical method used to describe variability amongobserved, correlated variables in terms of a potentially lower number offactors.

Factor model (also called herein member performance factor model)—amodel that includes one or more factors to describe a performance ofmembers and groups.

Group—a collection of members (e.g., machines, etc.). An exemplary groupis a group of machines at a factory. The composition of group may changeover time. The composition of group at a particular point in time orduring a particular period of time is determined by group memberweightings or group member exposures, defined below.

Member performance—the performance of a member S over a time period p.In some cases, the term “member performances” refers to a collection ofperformances over periods (p₁, . . . , p_(n)) in history H. As usedherein, history H refers to a set of time periods.

Group performance—a performance of a group P for a time period p. Insome cases, the term “group performances” refers to a collection ofperformances over periods in hi story H.

Factor performance—a performance of a particular factor F for a timeperiod p or a collection of performances of a particular factor F overperiods in history H.

Factor performances—a collection of performances of all factorsspecified by a particular factor model. The following is an example of adata structure containing factor performances for a particular point intime for the factors defined by an exemplary factor model represented instatistical programming notation:

-   -   factor.performances[1/31/2012]=    -   list(Group=1.0576, Milling Machines=−0.8514,    -   Lathes=0.0739, Machine Centers=1.2537, Drills=0.5742,    -   ForkLifts=−0.9453, Trucks=1.2373, Conveyor=0.7618,    -   Sorter=0.9826, Packager=0.0498,    -   Labeler=−0.0163, Size=0.6054, Value=0.7800,    -   Reference=0.0000)        In some cases, factor performances for a collection of periods        (p1, . . . , pn) in history H are expressed as a matrix where        each row contains factor performances for a time period p in        history H:

factor.performances[H] = matrix( Factory Milling Machines Lathes . . .Jan. 31, 2013 1.0576 −0.8514 0.0739 Feb. 28, 2013 3.2761 2.4519 4.0386 .. .)

Member factor exposures—a collection of terms representing therelationship between performances of a member and performances of eachfactor defined by a factor model. The following is an example of a datastructure containing factor exposures of a machine (MachineA) at aparticular point in time based on factors defined by an exemplary factormodel represented in statistical programming notation:

-   -   member.factor.exposures[MachineA, 2/28/2013]=    -   list(Group=0.9178, Milling Machines=0,    -   Lathes=0, Machine Centers=0, Drills=0,    -   ForkLifts=0, Trucks=0, Conveyor=0,    -   Sorter=0, Packager=0.0894,    -   Labeler=0, Size=0.2260, Value=0.0078,    -   Reference=1.0000)        In some cases, member factor exposures can be expressed for a        group P of members as a matrix where each row is member factor        exposures for a respective member S in the group P for a        period p. The following is an example of a data structure        containing factor exposures for members in the group P:

member.factor.exposures[P, Feb. 28, 2013] = matrix( Factory MilllingMachines Lathes . . . MachineA 1.0345 0 0 MachineB 1.1398 0 0 . . .)In some cases, the member factor exposures can be expressed for a memberover a collection of time periods (p1, . . . , pn) in history H as amatrix where each row is member factor exposures for a respective timeperiod p in H. The following is an example of a data structurecontaining factor exposures for a member FactoryA for time periods inhistory H:

member.factor.exposures[MachineA, H] = matrix( Factory Milling MachinesLathes . . . Jan. 31, 2013 0.9178 0 0 Feb. 28, 2013 0.9365 0 0 . . .)

Member performance attribution (also called herein decomposition)—Aprocess of describing a historical performance of a member in terms ofindependent variables (e.g., factors). In statistical programmingnotation, factor models define member performance r[S, p] for a member Sover a time period p as:

r[S,p]=sum(member.factor.exposures[S,p]*factor.performances[p])+member.residual.performance[S,p]

where member.factor.exposures[S, p] is a vector of terms capturing therelationship between the performances of S and each factor defined bythe factor model, factor.performances[p] is a vector of performancesover the time period p for each factor defined by the factor model andmember.residual.performance[S, p] is the portion of r[S, p] notattributable to the factors.

Compounded performance (also called herein multi-period performance)—arepresentative performance over a number of consecutive period. Forexample, a compounded performance for member S over periods (p1, . . .pn) in history H is determined from performances for member S (where aunit of performance is expressed in percents) in accordance with afollowing statistical programming notation:

compounded.r[S,H]=compounded(r[S,H])=(1+r[S,p1])* . . . *(1+r[S,pn])

In some embodiments, an active management skill is evaluated usingcompounded performances for predefined performance sources overpredefined periods.

Rate of performance (also called herein performance)—a change in a valueof a member or a group over a reference time period (e.g., one year).Typically, a performance is expressed as a fraction of a change in avalue of a member or a group over a reference time period over the valueof the member or the group. In some cases, the performance is expressedas a normalized performance over a standard time period (e.g., oneyear). In other words, an annualized rate of performance may be used(e.g., when the value of a member changes over 3 months is 1%, itsannualized performance of ˜4% may be used). In some embodiments, acontinuously compounded rate of performance is calculated for eachperiod and annualized.

Group member exposures—a collection of terms representing weights ofeach member within a group. In some embodiments, a group member exposurecorresponds to a ratio between a change in a performance value of agroup due to a change in a performance value of a member and the changein the performance value of the member at a corresponding point in time(or a corresponding time period). In some embodiments, a group memberexposure corresponds to a derivative of a performance value of a groupwith respect to a performance value of a respective member. An exampleof a data structure that represents group member exposures of a group Pat a particular point in time is as follows:

group.member.exposures[P,2/28/2013]=list(MachineA=5.00%,MachineB=10.00%, MachineC=10.00%, . . . )

Group factor exposure—exposure of a group to a particular factor at aspecific point in time.

group.factor.exposure[P,Factory,2/28/2013]=1.0345

Group factor exposures—a collection of terms representing therelationship between performances of a group and each factor defined bya factor model. In some embodiments, data representing group factorexposures has a same structure as data representing member factorexposures. The following is an example of a data structure containinggroup factor exposures for a particular point in time based on factorsdefined by an exemplary factor model represented in statisticalprogramming notation:

-   -   group.factor.exposures[P, 2/28/2013]=    -   list(Factory=1.5863, Milling Machines=0,    -   Lathes=−0.3785, Machine Centers=0, Drills=23.7415,    -   ForkLifts=0, Trucks=0, Conveyor=0,    -   Sorter=0, Packager=0.0894,    -   Labeler=0, Size=45.0733, Value=100.8918,    -   Reference=1.0000)        In some cases, group factor exposures are determined as        member-weighted sums of group member factor exposures (i.e.,        sums of member factor exposures multiplied by respective group        member exposures), or a matrix product of a member factor        exposures matrix and a group member exposures vector.        In some cases, group factor exposures for periods (p1, . . . ,        pn) in history H for a group P based on a set of factors (e.g.,        F1, F2, . . . ) is expressed in a matrix form as follows:

group.factor.exposures[P, H] = matrix( p1 p2 . . . F1group.factor.exposure[P, group.factor.exposure[P, F1, p1] F1, p2] F2group.factor.exposure[P, group.factor.exposure[P, F2, p1] F2, p2] . . .)

Member factor performance—a performance of member S for a time period pcontributed by factors. In some cases, the member factor performance isdetermined based on factor exposures of S and factor performances overp. In statistical programming notation, this can be expressed asfollows:

member.factor.performance[S,p]=sum(member.factor.exposures[S,p]*factor.performances[p])

Member residual performance (also called herein residual performance)—aperformance for member S for a time period p that is not attributable tofactors defined by a factor model, or a collection of such performancesover periods in history H. In some embodiments, the member residualperformance for member S in a time period p corresponds to a differencebetween a performance of member S for a time period p and a memberfactor performance of member S for the same time period p.

Member residual performances—a collection of residual performances formembers (S1, . . . , Sn) in a group P over a time period p.

Group factor performance—a performance of group P for a time period pcontributed by one or more factors F. In some cases, the group factorperformance is determined based on the group factor exposure of P to aparticular factor F for the time period p and the performance of theparticular factor F during the time period p, in which cases the groupfactor performance is sometimes referred to as the group factorperformance contribution of the particular factor F. In statisticalprogramming notation, this can be expressed as follows:

group.factor.performance.contribution[P,F,p]=group.factor.exposure[P,F,p]*factor.performance[F,p]

In some cases, group factor performance refers to a performance of agroup P for a time period p contributed by all factors defined by afactor model. In some cases, the group factor performance is determinedbased on the group factor exposures of P and factor performances for atime period p. In statistical programming notation, this can beexpressed as follows:

group.factor.performance[P,p]=sum(over all factorsF)(group.factor.exposures[P,p]*factor.performances[p])

Group residual performance (also called herein group idiosyncraticperformance)—a performance of a group P over a time period p that is notattributable to factors defined by a factor model. In some embodiments,the group residual performance for a group P corresponds to a differencebetween of a performance of group P for a time period p and group factorperformance of the group P for the same time period p.

Benchmark—a model group used to evaluate a relative performance of oneor more groups (e.g., all drilling contractors). As used herein, aperformance benchmark refers to a performance (e.g., a performance) of abenchmark.

Active management skill—one or both of: member selection skill andfactor timing skill.

Factor timing skill—an ability of an agent to consistently identifyex-ante a factor F and a period p from the aggregate history H such thatthe rate of performance for a factor F in a time period p exceeds a rateof performance for the factor F over history H.

Member selection skill (also called herein member picking skill)—theability of an agent to consistently identify a member S and period pfrom the aggregate history H such that the residual.performance[S, p]>0.

Factor timing performance (also called herein active factor performance,dynamic factor performance or factor timing performance)—a performancegenerated by varying a group factor exposure to a factor F over historyH in excess of a performance that would have been generated bymaintaining a constant (e.g., average) exposure to the factor F overhistory H. This may be expressed in statistical programming notation asfollows:

group.factor.dynamic.performance[P,F,p]=group.factor.performance.contribution[P,F,p]−group.factor.static.performance[P,F,p]

Member selection performance (also called herein member pickingperformance, idiosyncratic performance, group selection performance,group residual performance, or residual performance)—group residualperformance for group P over history H. Performance not attributable tothe group factor exposures, and instead attributable to, or indicativeof, skill in selecting the members of a group.

group.selection.performance[P,p]=sum(group.member.exposures[P,p]*member.residual.performances[P,p])

Active performance (also called herein dynamic performance)—A sum offactor timing performance and member selection performance. An activeperformance corresponds to a difference between a performance of group Pand a performance of benchmark B for a time period p of the history Hwhere benchmark B has group factor exposures that are the representativegroup factor exposures of P over the history H. Note that this issubstantially different from the definition of active performance in theexisting attribution method such as the Brinson model which may pick Bwithout regard for the similarity of the group factor exposures of B andrepresentative group factor exposures of P over the history H.

Passive performance (also called herein static performance and staticfactor performance)—A performance of benchmark B, where benchmark B hasgroup factor exposures that are the representative group factorexposures of P over history H.

Active factor exposures—differences in factor exposures between a groupP and a benchmark B for a time period p of the history H where benchmarkB has group factor exposures that are the representative group factorexposures of P over the history H.

Evidence of skill—statistical evidence that an agent possesses factortiming or member selection skill.

Confidence in skill—confidence level that an agent possesses factortiming or member selection skill, sometimes represented by a reliabilityvalue.

What is claimed is:
 1. A computer system, comprising: one or moreprocessors; and memory storing one or more programs for execution by theone or more processors, the one or more programs including instructionsfor: receiving, at the computer system, a request to quantify groupresidual performance associated with a first group; obtaining, at thecomputer system, historical group performance data for the first group;obtaining, at the computer system, historical group factor exposure datafor a plurality of predefined factors for the first group; obtaining, atthe computer system, historical factor performance data for theplurality of predefined factors for the first group; generating, at thecomputer system, historical group residual performance data for thefirst group in accordance with the historical group performance data forthe first group, the historical group factor exposure data for theplurality of predefined factors for the first group, and the historicalfactor performance data for the plurality of predefined factors for thefirst group, wherein a respective entry in the historical group residualperformance data represents member selection performance associated withthe first group; storing, at the computer system, the historical groupresidual performance data for the first group; and providing one or morevalues that represent the historical group residual performance data. 2.The computer system of claim 1, wherein the first group includes aplurality of members, the plurality of members including at least onemember of a first type and at least one member of a second type that isdifferent from the first type.
 3. The computer system of claim 1,wherein the first group includes a plurality of members, and all of theplurality of members in the first group are of a first type.
 4. Thecomputer system of claim 3, wherein: the first group is a respectivegroup in a plurality of groups; the plurality of groups includes asecond group that includes a plurality of members; and all of theplurality of members in the second group are of a second type that isdifferent from the first type.
 5. The computer system of claim 1,wherein: the first group is a respective group in a first set of groups,the first set of groups including a plurality of groups, and eachrespective group in the first set of groups including a plurality ofmembers; and the one or more programs further include instructions for:for each respective group in the first set other than the first group:obtaining, at the computer system, historical group performance data forthe respective group; obtaining, at the computer system, historicalgroup factor exposure data for a plurality of predefined factors for therespective group; obtaining, at the computer system, historical factorperformance data for the plurality of predefined factors for therespective group; and generating, at the computer system, respectivehistorical group performance data for the respective group in accordancewith the historical group performance data for the respective group, thehistorical group factor exposure data for the plurality of predefinedfactors for the respective group, and the historical factor performancedata for the plurality of predefined factors for the respective group,wherein a respective entry in the respective historical group residualperformance data for the respective group represents member selectionperformance associated with the respective group.
 6. The computer systemof claim 5, wherein the one or more programs further includeinstructions for: identifying a subset of groups in the first set basedat least in part on the respective historical group residual performancedata for each respective group in the first set; generating a second setof groups that consists of the subset of groups; and providing one ormore values that represent the second set of groups.
 7. The computersystem of claim 6, wherein identifying the subset of groups in the firstset is performed without regard to whether a respective group includesmembers of a particular type.
 8. The computer system of claim 6, whereinall of the members in the first set of groups are of a first type. 9.The computer system of claim 6, wherein: a respective entry in therespective historical group residual performance data for a respectivesubset represents member selection performance associated with therespective subset during a respective time period in a predefined set oftime periods; and identifying the subset of groups in a respective setincludes identifying a predetermined number of groups having the highestrespective entries in the respective historical group residualperformance data during a particular time period in the predefined setof time periods.
 10. The computer system of claim 9, wherein eachrespective time period in the predefined set of time periods spans onemonth.
 11. The computer system of claim 6, wherein: a respective entryin the respective historical group residual performance data for arespective subset represents member selection performance associatedwith the respective subset during a respective time period in apredefined set of time periods; and identifying the subset of groups ina respective set includes identifying groups having respective entriesin the respective historical group residual performance data thatsatisfy a predefined threshold during a particular time period in thepredefined set of time periods.
 12. The computer system of claim 6,wherein the one or more programs include instructions for: determining,for each respective group, a respective reliability value that indicatesreliability of the respective historical group residual performance dataof the respective subset over a predefined set of time periods; whereinidentifying the subset of groups in a respective set includesidentifying a predetermined number of groups having the highestrespective reliability values.
 13. The computer system of claim 12,wherein the one or more programs include instructions for: identifying adistribution of values associated with the reliability values;determining a first number of reliability values that are within a firstrange within the distribution of values; and determining whether thefirst number of reliability values exceeds a threshold number ofreliability values by at least a predefined amount; wherein identifyingthe subset of groups in a respective set is further based on adetermination that the first number of reliability values exceeds thethreshold number of reliability values by at least the predefinedamount.
 14. The computer system of claim 6, wherein the one or moreprograms include instructions for: determining, for each respectivegroup, a respective reliability value that indicates reliability of therespective historical group residual performance data of the respectivesubset over a predefined set of time periods; wherein identifying thesubset of groups in a respective set includes identifying groups havingrespective reliability values that satisfy a predefined threshold. 15.The computer system of claim 6, wherein: the respective historical groupresidual performance data for a respective group includes: respectivehistorical group residual performance data for the respective group overa first set of time periods; and respective historical group residualperformance data for the respective group over one or more subsequentsets of time periods, wherein the one or more subsequent sets of timeperiods occur later in time than the first set of time periods; and theone or more programs include instructions for: determining, for eachrespective group, one or more respective autocorrelation valuesindicating a degree of autocorrelation between the respective historicalgroup residual performance data for the respective group during thefirst set of time periods and the respective historical group residualperformance data for the respective group during the one or moresubsequent sets of time periods; wherein identifying the subset ofgroups in a respective set is further based at least in part on adetermination that autocorrelation values for the subset of groupssatisfy predefined autocorrelation criteria.
 16. The computer system ofclaim 5, wherein: all of the members in the first set of groups are of afirst type; a third set of groups includes a plurality of groups, eachrespective group in the third set of groups including a plurality ofmembers; all of the members in the third set of groups are of a secondtype that is different from the first type; and the one or more programsfurther include instructions for: for each respective group in the thirdset: obtaining, at the computer system, historical group performancedata for the respective group; obtaining, at the computer system,historical group factor exposure data for a plurality of predefinedfactors for the respective group; obtaining, at the computer system,historical factor performance data for the plurality of predefinedfactors for the respective group; and generating, at the computersystem, respective historical group residual performance data for therespective group in accordance with the historical group performancedata for the respective group, the historical group factor exposure datafor the plurality of predefined factors for the respective group, andthe historical factor performance data for the plurality of predefinedfactors for the respective group, wherein a respective entry in thehistorical group residual performance data for the respective grouprepresents member selection performance associated with the respectivegroup; identifying a subset of groups in the first set based at least inpart on the respective historical group residual performance data foreach respective group in the first set; identifying a subset of groupsin the third set based at least in part on the respective historicalgroup residual performance data for each respective group in the thirdset; generating a fourth set of groups that includes the identifiedsubset of groups in the first set and the identified subset of groups inthe third set; and providing one or more values that represent thefourth set of groups.
 17. The computer system of claim 1, wherein theone or more programs further include instructions for: obtaining, foreach member in the first group, historical use data for the respectivemember; and generating historical group member use data for the firstgroup in accordance with the respective historical use timing data foreach respective member in the first group; wherein: obtaining thehistorical group performance data for the first group includes:obtaining historical member performance data for the first group; andgenerating the historical group performance data for the first group inaccordance with the historical group member use data for the first groupand the historical member performance data for the first group; andobtaining the historical group factor exposure data for the plurality ofpredefined factors for the first group includes: obtaining historicalmember factor exposure data for the first group; and generating thehistorical group factor exposure data for the plurality of predefinedfactors for the first group in accordance with the historical groupmember use data for the first group and the historical member factorexposure data for the first group.
 18. The computer system of claim 1,wherein obtaining the historical group factor exposure data for theplurality of predefined factors for the first group includes: obtaininginitial historical group factor exposure data for the plurality ofpredefined factors for the first group; generating an aggregatehistorical group factor exposure value for the plurality of predefinedfactors for a group including all possible members, based on the initialhistorical group factor exposure data for the plurality of predefinedfactors for the first group; determining an offset between the aggregatehistorical group factor exposure value for the plurality of predefinedfactors for the group including all possible members and an expectedaggregate historical group factor exposure value; and generating thehistorical group factor exposure data for the plurality of predefinedfactors for the first group based on the initial historical group factorexposure data for the plurality of predefined factors for the firstgroup and the determined offset.
 19. The computer system of claim 1,wherein: obtaining, at the computer system, the historical groupperformance data for the first group includes obtaining historical groupperformance data for at least a first subgroup in the first group and asecond subgroup in the first group, wherein the first subgroupcorresponds to a first time period, and wherein the second subgroupcorresponds to a second time period distinct from the first time period.20. A method for quantifying performance of a group, comprising:receiving, at a computer system, a request to quantify group residualperformance associated with a first group; obtaining, at the computersystem, historical group performance data for the first group;obtaining, at the computer system, historical group factor exposure datafor a plurality of predefined factors for the first group; obtaining, atthe computer system, historical factor performance data for theplurality of predefined factors for the first group; generating, at thecomputer system, historical group residual performance data for thefirst group in accordance with the historical group performance data forthe first group, the historical group factor exposure data for theplurality of predefined factors for the first group, and the historicalfactor performance data for the plurality of predefined factors for thefirst group, wherein a respective entry in the historical group residualperformance data represents member selection performance associated withthe first group; storing, at the computer system, the historical groupresidual performance data for the first group; and providing one or morevalues that represent the historical group residual performance data.